21 Mar 2019
Django community aggregator: Community blog posts
Best way to count distinct indexed things in PostgreSQL
`SELECT COUNT(*) FROM (SELECT DISTINCT my_not_unique_indexed_column FROM my_table) t`
21 Mar 2019 10:13pm GMT
Coding for Time Zones & Daylight Saving Time — Oh, the Horror
In this post, I review some reasons why it's really difficult to program correctly when using times, dates, time zones, and daylight saving time, and then I'll give some advice for working with them in Python and Django. Also, I'll go over why I hate daylight saving time (DST).
TIME ZONES
Let's start with some problems with time zones, because they're bad enough even before we consider DST, but they'll help us ease into it.
Time Zones Shuffle
Time zones are a human invention, and humans tend to change their minds, so time zones also change over time.
Many parts of the world struggle with time changes. For example, let's look at the Pacific/Apia time zone, which is the time zone of the independent country of Samoa. Through December 29, 2011, it was -11 hours from Coordinated Universal Time (UTC). From December 31, 2011, Pacific/Apia became +13 hours from UTC.
What happened on December 30, 2011? Well, it never happened in Samoa, because December 29, 23:59:59-11:00 is followed immediately by December 31, 0:00:00+13:00.
| Date | Time | Zone | Date | Time | Zone |
|---|---|---|---|---|---|
| 2011-12-29 | 23:59:59 | UTC-11 | 2011-12-30 | 10:59:59 | UTC |
| 2018-12-31 | 00:00:00 | UTC+13 | 2011-12-30 | 11:00:00 | UTC |
That's an extreme example, but time zones change more often than you might think, often due to changes in government or country boundaries.
The bottom line here is that even knowing the time and time zone, it's meaningless unless you also know the date.
Always Convert to UTC?
As programmers, we're encouraged to avoid issues with time zones by "converting" times to UTC (Coordinated Universal Time) as early as possible, and convert to the local time zone only when necessary to display times to humans. But there's a problem with that.
If all you care about is the exact moment in the lifetime of the universe when an event happened (or is going to happen), then that advice is fine. But for humans, the time zone that they expressed a time in can be important, too.
For example, suppose I'm in North Carolina, in the eastern time zone, but I'm planning an event in Memphis, which is in the central time zone. I go to my calendar program and carefully enter the date and "3:00 p.m. CST". The calendar follows the usual convention and converts my entry to UTC by adding 6 hours, so the time is stored as 9:00 p.m. UTC, or 21:00 UTC. If the calendar uses Django, there's not even any extra code needed for the conversion, because Django does it automatically.
The next day I look at my calendar to continue working on my event. The event time has been converted to my local time zone, or eastern time, so the calendar shows the event happening at "4:00 p.m." (instead of the 3:00 p.m. that it should be). The conversion is not useful for me, because I want to plan around other events in the location where the event is happening, which is using CST, so my local time zone is irrelevant.
The bottom line is that following the advice to always convert times to UTC results in lost information. We're sometimes better off storing times with their non-UTC time zones. That's why it's kind of annoying that Django always "converts" local times to UTC before saving to the database, or even before returning them from a form. That means the original timezone is lost unless you go to the trouble of saving it separately and then converting the time from the database back to that time zone after you get it from the database. I wrote about this before.
By the way, I've been putting "convert" in scare quotes because talking about converting times from one time zone to another carries an implicit assumption that such converting is simple and loses no information, but as we see, that's not really true.
DAYLIGHT SAVING TIME
Daylight saving time (DST) is even more of a human invention than time zones.
Time zones are a fairly obvious adaptation to the conflict between how our bodies prefer to be active during the hours when the sun is up, and how we communicate time with people in other parts of the world. Historical changes in time zones across the years are annoying, but since time zones are a human invention it's not surprising that we'd tweak them every now and then.
DST, on the other hand, amounts to changing entire time zones twice every year. What does US/eastern time zone mean? I don't know, unless you tell me the date. From January 1, 2018 to March 10, 2018, it meant UTC-5. From March 11, 2018 to November 3, 2018, it meant UTC-4. And from November 4, 2018 to December 31, 2018, it's UTC-5 again.
But it gets worse. From Wikipedia:
The Uniform Time Act of 1966 ruled that daylight saving time would run from the last Sunday of April until the last Sunday in October in the United States. The act was amended to make the first Sunday in April the beginning of daylight saving time as of 1987. The Energy Policy Act of 2005 extended daylight saving time in the United States beginning in 2007. So local times change at 2:00 a.m. EST to 3:00 a.m. EDT on the second Sunday in March and return at 2:00 a.m. EDT to 1:00 a.m. EST on the first Sunday in November.
So in a little over 50 years, the rules changed 3 times.
Even if you have complete and accurate information about the rules, daylight saving time complicates things in surprising ways. For example, you can't convert 2:30 a.m. March 11, 2018. in US/eastern time zone to UTC, because that time never happened - our clocks had to jump directly from 1:59:59 a.m. to 3:00:00 a.m. See below:
| Date | Time | Zone | Date | Time | Zone |
|---|---|---|---|---|---|
| 2018-03-11 | 1:59:59 | EST | 2018-03-11 | 6:59:59 | UTC |
| 2018-03-11 | 3:00:00 | EDT | 2018-03-11 | 7:00:00 | UTC |
You can't convert 1:30 a.m. November 4, 2018, in US/eastern time zone to UTC either, because that time happened twice. You would have to specify whether it was 1:30 a.m. November 4, 2018 EDT or 1:30 a.m. November 4, 2018 EST:
| Date | Time | Zone | Date | Time | Zone |
|---|---|---|---|---|---|
| 2018-11-04 | 1:00:00 | EDT | 2018-11-04 | 5:00:00 | UTC |
| 2018-11-04 | 1:30:00 | EDT | 2018-11-04 | 5:30:00 | UTC |
| 2018-11-04 | 1:59:59 | EDT | 2018-11-04 | 5:59:59 | UTC |
| 2018-11-04 | 1:00:00 | EST | 2018-11-04 | 6:00:00 | UTC |
| 2018-11-04 | 1:30:00 | EST | 2018-11-04 | 6:30:00 | UTC |
| 2018-11-04 | 1:59:59 | EST | 2018-11-04 | 6:59:59 | UTC |
Advice on How to Properly Manage datetimes
Here are some rules I try to follow.
When working in Python, never use naive datetimes. (Those are datetime objects without timezone information, which unfortunately are the default in Python, even in Python 3.)
Use the pytz library when constructing datetimes, and review the documentation frequently. Properly managing datetimes is not always intuitive, and using pytz doesn't prevent me from using it incorrectly and doing things that will provide the wrong results only for some inputs, making it really hard to spot bugs. I have to triple-check that I'm following the docs when I write the code and not rely on testing to find problems.
Let me strengthen that even further. It is not possible to correctly construct datetimes with timezone information using only Python's own libraries when dealing with timezones that use DST. I must use pytz or something equivalent.
If I'm tempted to use datetime.replace, I need to stop, think hard, and find another way to do it. datetime.replace is almost always the wrong approach, because changing one part of a datetime without consideration of the other parts is almost guaranteed to not do what I expect for some datetimes.
When using Django, be sure USE_TZ = True. If Django emits warnings about naive datetimes being saved in the database, treat them as if they were fatal errors, track them down, and fix them. If I want to, I can even turn them into actual fatal errors; see this Django documentation.
When processing user input, consider whether a datetime's original timezone needs to be preserved, or if it's okay to just store the datetime as UTC. If the original timezone is important, see this post I wrote about how to get and store it.
Conclusion
Working with human times correctly is complicated, unintuitive, and needs a lot of careful attention to detail to get right. Further, some of the oft-given advice, like always working in UTC, can cause problems of its own.
21 Mar 2019 9:45pm GMT
Backup & Load Live Heroku PostgreSQL Database to Local Django Project
I've used [Django fixtures](ht...
21 Mar 2019 7:04am GMT
Where to Learn Django in 2019
I quite often get asked for resources on where to learn Django. Here's a list of the places I know!
The Deep End
First, a warning: there's a lot of stuff to know. Learning Django with zero web or programming knowledge is diving in at the deep end. If you've never done anything like it, you'll be learning about Python, databases, HTML, CSS, browsers, Git, Django, and more, all at the same time.
I'd recommend knowing how to build a static website before attempting to learn Django. There are many ways to do this, for free tutorials check out W3Schools or Learn Web Development from Mozilla.
But if you think you're ready to begin, let's continue.
Written Tutorials
First and foremost is the official "Start" page which links to several things, including the official tutorial. A warning: the tutorial is quite comprehensive, and is not really designed for zero-experience beginners. It also has some blind spots, for example it does not feature any advice on installing Python, or deploying your website to the actual internet.
The Django Girls tutorial is a full guide to creating and deploying a blog for free. This organization uses it to teach weekend workshops to women around the world, so it's pretty battle-tested. I'd recommend this as easiest starting point for most.
I've also recently learned that Mozilla have a very complete tutorial in their learning area. They use a lot of Django so I expect this should be quite up to date.
If you're building an API, Django Rest Framework is a popular Django package to help this. It has a quickstart tutorial for getting started.
Books
I can't say I've read many of the books out there targeted at beginners, so I don't want to endorse any in particular at the cost of the others. The Awesome Django project keeps a list of known certified "awesome" books though.
However I can say that I've enjoyed every version of Two Scoops of Django. It's full of tips more suitable for those with a bit of experience, so if you're just starting today, earmark it as an ice-cream themed present for yourself in the future.
Videos
I also can't say I've watched a beginner's video tutorial on Django, as I normally prefer to read. There are a lot listed on Awesome Django though.
What I have seen is a lot of conference talks, and most of these make their way to the web. Yes, you can sit on your sofa and watch talks that hundreds of developers spent lots of time and money travelling from around the world to see, all for free. The internet sounds absurd sometimes, but that really is the world we live in!
Awesome Django has a list of conference YouTube channels. Check them out for beginner's talks or stories, or even just the most popular videos.
There are also tons of talks about Django from other conferences, meet-ups, and various events around the world, all right there on YouTube, Vimeo, or other sites. It's worth searching around if there's a specific topic you want to learn more about.
Podcasts
The only active Django-specific podcast I know of is the recently started Django Chat, hosted by Will Vincent and current Django fellow Carlton Gibson. Episode 2 is "How to Learn Django" which is obviously relevant for this post, check it out!
Documentation
The Django documentation can be a bit big and scary to begin with, but it's really the only reliable guide on things work. It's available partially translated to other languages too. At the time of writing the Transifex project shows French, Indonesian, and Japanese as the top three languages. It's never 100% translated because translators are playing catch-up with the ongoing changes.
(An easy way to start contributing is to translate to your native language whilst you read through the documentation. See Localizing Django.)
It's also worth downloading a copy for local reference when you don't have an internet connection. See the "Download" link on the right sidebar of the docs front page.
I personally use a Mac App called Dash that gives quick offline access to all kinds of programming documentation. There's a free clone of it at DevDocs.io - and it has an offline mode as well, which is really cool.
Whilst you're into reading documentation, I'd recommend Kristian Glass's Unofficial Opinionated FAQ as an example of the kind of choices that Django doesn't force on you but are probably good ideas. I agree with everything in that post.
Meetups
As one of the organisers for the London Django Meetup I'd like to highly recommend going to a local group. Even if you haven't managed to install Django, going to meet some people in your area can inspire you, give you pointers, and you might even find a co-learner or mentor.
Most meetups register on Meetup.com these days, and they have a page of Django meetups. At the time of writing there are 422 meetups in 67 countries, with a total of 430,065 members - that's more than the population of Brunei!
Blogs
Following the Django community is a great way of finding new avenues of learning. I follow the "Community blog posts" feed from the community page in my feed reader ( Old Reader).
If you don't have a feed reader, don't worry, they're kind of old fashioned. You can also set up getting feed items as emails (or SMS's, or pretty much anything else) with a free account on Zapier using this integration (fun fact, Zapier is written in Django!).
Some organizational blogs that spring to mind with good historical content worth reading are:
(I decided not to include individuals here because there are just so many! Maybe one day I'll post all the Django-related blogs I subscribe to.)
Every Day is a Learning Day
A closing note: Every programmer, even the most expert, is continuously learning. If you feel overwhelmed, just remember we are all taking it one day at a time. Before you know it, you'll be building useful applications to provide value to those around you.
Hope this bunch of resources helps you start or improve your Django skills.
-Adam
P.S. Thanks to Carlton Gibson, James Spurin, Neil Bachelor, and Ngazentungue Muheue for reviewing the post.
21 Mar 2019 5:00am GMT
20 Mar 2019
Django community aggregator: Community blog posts
Convert SVG Icons To CSS Webfonts and Deploy to CDN
If you're like me, you've undo...
20 Mar 2019 7:10am GMT
19 Mar 2019
Django community aggregator: Community blog posts
Create a Standalone React App
This post will answer the foll...
19 Mar 2019 7:53pm GMT
10 Common Design Mistakes… and How to Avoid Them
Design work is complex business. It's far too easy to mess up. To create high-quality products, all of the small details are crucially important. Each detail must be taken into account. It can be easy to get lost in the details and lose sight of the big picture. That's why there are several incredibly common user interface (UI) design mistakes and weaknesses. The post 10 Common Design Mistakes… and How to Avoid Them appeared first on Distillery.
19 Mar 2019 2:14pm GMT
17 Mar 2019
Django community aggregator: Community blog posts
Hello Linux: Celery & Supervisor
17 Mar 2019 9:44pm GMT
Hello Linux: Install Redis
17 Mar 2019 9:14pm GMT
16 Mar 2019
Django community aggregator: Community blog posts
Tips for Testing Django Views
Views are probably the most essential part of a Django application. They provide an interface to your application for users or other external applications. It only makes sense that these need to be well-tested. At first, it can seem difficult and/or tedious to write unit tests for them. However, by decoupling your business logic from … Continue reading Tips for Testing Django Views
The post Tips for Testing Django Views appeared first on concise coder.
16 Mar 2019 10:27pm GMT
15 Mar 2019
Django community aggregator: Community blog posts
Using Multiple SSH Keys
Every time you use a new host,...
15 Mar 2019 6:33pm GMT
Using SSH & Creating SSH Keys
Using a Secure Shell (`ssh`) w...
15 Mar 2019 5:33pm GMT
Hello Linux: Custom Domain & Https with Let's Encrypt
15 Mar 2019 12:09am GMT
14 Mar 2019
Django community aggregator: Community blog posts
Hello Linux: Nginx & UFW Firewall
14 Mar 2019 11:24pm GMT
Hello Linux: Gunicorn & Supervisor
14 Mar 2019 10:54pm GMT
Hello Linux: PostgreSQL on Live Linux Server
14 Mar 2019 9:24pm GMT