Do you guys know how to use list comprehensions? I didn't know about them until using Python for over a year.
even_numbers = [x for x in range(0,100) if x % 2 == 0]
You can also do set and dict comprehensions.
Other cool stuff some people may not know about: - lambda expressions - generators and generator expressions - decorators - the "with" statement - comparison operators can be chained, e.g. 1 < x <= 4 translates the same way in Python that it does in maths.
There's probably more stuff that people might not know, but that's what comes to mind.
Yeah... I mostly dabble in Python as I'm primarily a C/C++ programmer, but Python is my go-to language when I need to do any sort of scripting (I spent quite a bit of time writing a test suite for a feature of the C++ program I work on in Python -- works great). It's definitely always a nice surprise when I notice Python has a neat new feature that makes those scripts easier.
My main complaint with Python (and Scott knows about this) is that apparently its standard libraries are crap (at least according to Scott). If you want to find a library to do something, you're better off finding some third party library than using what ships with the language. This is pretty much the polar opposite of C/C++, where the standard libraries (or semi-standard ones, such as Boost) pretty much contain the best overall (though maybe not for specific use cases) implementations of those features out there.
Personally, I think that if your standard library is crap compared to some random third party library, then you should either rework your standard library to make it better or incorporate the third party stuff into a new revision of your standard library.
I know one tends to use plenty of third party libraries in Python, but generally this is to do things that aren't in the standard libraries, not things that are done badly in them.
I had an issue with Python's process spawning libraries while writing that aforementioned test suite. I used Python more or less as "glue code" to connect multiple C/C++ command line programs together. Basically, I used it like Bash on steroids, with real functions, classes, my own custom libraries, XML parsers (my suite used an XML config file to specify its options), and so on. I complained about the issue on Twitter or Google+ or something, and Scott was basically like, "It's your own damned fault for using Python's built-in libraries instead of finding a third-party library." He didn't give any specific examples, though, if I remember correctly.
I agree that using non-standard libraries in any language is perfectly acceptable for functionality not available in the standard libraries, no matter the language. However, the standard library should be the canonical, reliable, preferred choice for functionality that's available inside it.
There are a lot of reasons the Python standard library sucks. And frankly, it's so easy to use get extra libraries, it really doesn't matter. Heck, it's even easier now that the setuptools debacle has been resolve and pypi has been moved to a CDN.
Breaking old programs that use the crappy standard library is one reason the Python standard stuff sucks. But the real reason is actually a positive. Updating Python is slow. Patching the language itself is not something to be taken lightly.
The reason that third party libraries are so good is because the actively developed ones are iterated rapidly. Even the bigger ones like Django get updates very very often. This means that the people who develop the language itself are able to also iterate more rapidly because they are focusing only on core language issues, and not all sorts of other things like image processing (PIL), http (wsgi), or package management(setuptools).
The only major problem with this model is that Python tutorials do not teach this. They teach about Python. They don't teach about all these third party libraries, how to install them, how to use them. It takes maybe 15-20 minutes to learn how to use setutools/pip/virtualenv. But then you have to know the "magic words." Magic words are a problem across all high level programming languages these days, but especially with Python.
Let's say you do a Python tutorial You want to make a video game. You need to make graphics show up on the screen. Even if you know pip/virtualenv, what library do you use? PySDL? PyGame? How did you even know about PySDL? Someone had to tell you the magic words "PySDL". Or at least those magic words had to show up in a Google search. And if they did, how do you know that's the right choice? How do you know there isn't some other better library out there? The official Python website says nothing about it. PyPi just lists every package, and gives them ratings. But a search for graphics gives you http://cgkit.sourceforge.net/ as the first result. Is that really what you want?
That's the real problem. You have to at least lurk in the Python community to know about the existence of the specific package that is going to solve each problem. I think I remember Luke once wrote a web scraper with urllib instead of scrapy. All he had to do was know the magic word "scrapy", but how would he possibly know it? If it was in the standard library, it would be on python.org, and it could be found.
The solution that I think has been discussed, but not implemented, is pretty simple. Python.org should not include these things in STL, but should give their formal blessing to some third party libraries. Those packages should be referenced in documentation on python.org. Their should be a list somewhere of packages that are best in class. So you can easily search for what problem you are trying to solve, and it will tell you which packages are going to get you most of the way there. Also, setuptools/pip/virtualenv, and how they work, need to be taught very early in Python tutorials.
C++ does something kind of similar to that with Boost. In fact, Boost itself works as an incubator for new features for the C++ STL as many of its developers also work on the STL and are on the C++ standards committee. A perfect example of this would be the C++ reference counted, shared smart pointer. It's in the most recent C++ standards but first appeared in Boost as its shared_ptr class. For what it's worth, Boost also still contains a shared_ptr implementation, but I think by default it will use the one that ships with your compiler if available.
I'd be fine if Python had a rock solid standard library for common functionality and simply pointed to equivalents for something like Boost.
Rapid iteration isn't always a good thing either, depending on what's going on with the specific library. There's something to be said for a library that's used by nearly everyone using your language (whether standard or semi-standard like Boost). All those users banging on it in different ways will help expose lots of bugs. The problem with rapid iteration is that often many new bugs could be inserted -- but I admit this can be mitigated by simply doing something like having separate stable and development branches where the stable branch doesn't change frequently except for bug fixes vs. the dev branch that contains all the new hotness but may be buggier. Again, though, if there was a "Python Boost" that nearly everyone using Python used, this would also accomplish the goal or having something that has tons of users banging on it to expose bugs.
Still, that doesn't mean you shouldn't have a STL of some sort. The purpose of an STL isn't necessarily to have the hottest new features, but to have stuff that's rock-stable, always available, and doesn't need to be rapidly iterated. The C++ std::string class, for example, hasn't really changed much since the 90's, and, frankly, it doesn't need to. Any bugs in it have been worked out long ago and it does a perfect fine job of providing a string type to your code. If Python's STL is broken in some way and not being properly maintained, then either the broken feature needs to be removed from the STL (as it fails the test of being rock-stable) or, dammit, it needs to be fixed and made rock-stable.
Maybe the problem is that Python's STL is simply too far-reaching in functionality to truly be an STL. C++'s provides core functionality like strings, I/O, memory management, containers (lists, vectors, etc.), and so on. If you need anything more exotic (regular expressions, fancy numeric types, threads, etc.), then you usually end up turning to Boost (although some of these Boost features, like threads, are showing up in the newest versions of the C++ standard library as they are now considered core features of modern programs). Granted, Python doesn't necessarily need features like these in its STL as they're built-in or unnecessary to the language itself, but a similar rule of thumb may not be a bad idea. Having an XML parser in the Python standard library is certainly useful, but if it's just going to sit there and rot and only be used by a handful of the myriad of Python apps in the world, maybe it shouldn't be in the STL.
I will say this, though -- ease of use of installing libraries via pip or whatnot is fine if you're writing a serious Python app but even that is a pain in the ass if you simply writing some basic scripts that you want to distribute where the only requirement is that you have a working Python installation. If the test scripts I wrote were only going to be used by me, then yeah, I'd have no problem with installing whatever other random Python package in order to make them work. However, these scripts were meant to be used by my co-workers in addition to myself and I didn't want to make them have to jump through all sorts of hoops just to get those scripts running on their test boxes.
I was going to mention, but you already did, that C++ and Python are quite different in what constitutes libraries. In C++ you need a library just do something like a Linked List. In Python you only need a library to do something like a gui, which is also true for C++.
As for distribution of Python software with external requirements, you have a point, but there are many solutions to that.
The best solution is to simply use setuptools! Then instead of giving your co-workers code on a USB stick, they simply do
pip install louscode
DONE!
Oh, you don't want to share your code publicly? Well, you have an internal way to host your code right? Perhaps your git repository? Tell your co-workers to do this:
Now, if you want to educate your co-workers are bit, like mine are, you can simply include a requirements.txt file in your project. Then it's up to your co-workers to know that they have to do this:
Four commands, one of which is cd. Not too shabby if I say so myself. The mkvirtualenv is even optional, if you don't care about isolating the libraries (but you do).
Yet another option is to use your system package manager, like apt! Create a .deb with proper dependencies. Then when your co-workers apt-get install it, it will automatically also install any python-XXX packages that you specified, and all the major ones are usually available.
Now, there are indeed some times that this can screw up. For example, let's say you are doing image processing, and PIL(Pillow is better) is a requirement. Also, you are going to be processing jpegs. If all your co-workers do is pip install pillow, jpeg support will not be there. It will error at runtime when they try to edit a jpeg. This is because they need to have libjpeg-dev already installed via the OS. Most Python packages do not have C-libraries as requirements, but a few do. The ones I encounter most often are usually just wrappers for C libraries in the first place. Examples are the aforementioned PIL, numpy/scipy, DB libraries like MySQL-Python and psycopg2 (postgres), pylibmc (memcached), etc. If all you are doing is sharing scripts with co-workers it is much more likely that the libraries you are using are going to be pure Python, and will not have this issue.
It's slightly more complicated than that. The scripts have to be able to run on the same OS we ship to our customers and said OS image doesn't include pip (though it includes Python). Adding pip or anything else to the customer OS image is not a trivial thing. They also have to run on Windows (that's part of the reason why I selected Python -- cross-platform capabilities!), although pip does run on Windows. In theory, though, I could probably put together an RPM just for testing purposes (our customer OS distro is RPM-based), at least for Linux testing purposes. For Windows, hmm, I'd have to learn MSI or perhaps see if Nullsoft's Super-Pimp Installer is still around.
For distribution, you are right in that we do keep it in our source code repository (we don't use git, but the exact tool doesn't really matter in this case), so it's not like I'm handing out USB sticks.
The question is whether all the extra effort to bundle it with the non-standard libraries is worth it. Given how I figured out how to cope with the bug that was causing those script failures early on, it doesn't seem like it would be worth it, at least for now.
As for adding pip to the OS image, maybe you won't have to. easy_install (setuptools) has a very good chance of already being there. pip is really just a better easy_install that allows you do things like use requirements.txt files and install direct from Git repos. You can just easy_install everything. You can even just include setup.py files and do python setup.py
To be fair, I'd probably be more in the community if Python was something I did full-time, as opposed to something I did occasionally when I need to hack a script together.
I should write "secret guide to Python." It wouldn't be that long. It would introduce people who took a Python tutorial into the world of using packages and such.
I should write "secret guide to Python." It wouldn't be that long. It would introduce people who took a Python tutorial into the world of using packages and such.
Please, please do this! Assume people don't know any code words, just the basic Python documentation.
I sort of did part of it already, but it was before I was allowed to make YouTube videos longer than 10-15 minutes, and some things have changed slightly with time.
List comprehensions are super common in C# (Linq). I didn't know you could do them in python but I hardly ever use python except for writing dirty scripts to usually process some data real quick. I've always been surprised that Java doesn't seem to have anything for it (at least to my knowledge). Then again I'm always surprised at the lack of features Java has sometimes.
So, in C++ I have a class Vote which has a send function to send the vote over the network. This send function in the class Vote, makes use of the send function from sys/socket.h. When I compile, I get errors that basically say it's using Vote::send rather than sys/socket.h::send within Vote::send. How do I resolve this name conflict?
Edit: stackoverflow got my back. The answer? Refer to sys/socket.h::send as ::send.
I'm trying out C#, still getting all the syntax together, it isn't intuitive for me but bare in mind I'm coming at it from a noob perspective (have only been doing console programs that do various maths and display tasks).
I will start heavy on Java and Python in a couple of weeks.
Comments
Other cool stuff some people may not know about:
- lambda expressions
- generators and generator expressions
- decorators
- the "with" statement
- comparison operators can be chained, e.g. 1 < x <= 4 translates the same way in Python that it does in maths.
There's probably more stuff that people might not know, but that's what comes to mind.
My main complaint with Python (and Scott knows about this) is that apparently its standard libraries are crap (at least according to Scott). If you want to find a library to do something, you're better off finding some third party library than using what ships with the language. This is pretty much the polar opposite of C/C++, where the standard libraries (or semi-standard ones, such as Boost) pretty much contain the best overall (though maybe not for specific use cases) implementations of those features out there.
Personally, I think that if your standard library is crap compared to some random third party library, then you should either rework your standard library to make it better or incorporate the third party stuff into a new revision of your standard library.
I know one tends to use plenty of third party libraries in Python, but generally this is to do things that aren't in the standard libraries, not things that are done badly in them.
I agree that using non-standard libraries in any language is perfectly acceptable for functionality not available in the standard libraries, no matter the language. However, the standard library should be the canonical, reliable, preferred choice for functionality that's available inside it.
Breaking old programs that use the crappy standard library is one reason the Python standard stuff sucks. But the real reason is actually a positive. Updating Python is slow. Patching the language itself is not something to be taken lightly.
The reason that third party libraries are so good is because the actively developed ones are iterated rapidly. Even the bigger ones like Django get updates very very often. This means that the people who develop the language itself are able to also iterate more rapidly because they are focusing only on core language issues, and not all sorts of other things like image processing (PIL), http (wsgi), or package management(setuptools).
The only major problem with this model is that Python tutorials do not teach this. They teach about Python. They don't teach about all these third party libraries, how to install them, how to use them. It takes maybe 15-20 minutes to learn how to use setutools/pip/virtualenv. But then you have to know the "magic words." Magic words are a problem across all high level programming languages these days, but especially with Python.
Let's say you do a Python tutorial You want to make a video game. You need to make graphics show up on the screen. Even if you know pip/virtualenv, what library do you use? PySDL? PyGame? How did you even know about PySDL? Someone had to tell you the magic words "PySDL". Or at least those magic words had to show up in a Google search. And if they did, how do you know that's the right choice? How do you know there isn't some other better library out there? The official Python website says nothing about it. PyPi just lists every package, and gives them ratings. But a search for graphics gives you http://cgkit.sourceforge.net/ as the first result. Is that really what you want?
That's the real problem. You have to at least lurk in the Python community to know about the existence of the specific package that is going to solve each problem. I think I remember Luke once wrote a web scraper with urllib instead of scrapy. All he had to do was know the magic word "scrapy", but how would he possibly know it? If it was in the standard library, it would be on python.org, and it could be found.
The solution that I think has been discussed, but not implemented, is pretty simple. Python.org should not include these things in STL, but should give their formal blessing to some third party libraries. Those packages should be referenced in documentation on python.org. Their should be a list somewhere of packages that are best in class. So you can easily search for what problem you are trying to solve, and it will tell you which packages are going to get you most of the way there. Also, setuptools/pip/virtualenv, and how they work, need to be taught very early in Python tutorials.
I'd be fine if Python had a rock solid standard library for common functionality and simply pointed to equivalents for something like Boost.
Rapid iteration isn't always a good thing either, depending on what's going on with the specific library. There's something to be said for a library that's used by nearly everyone using your language (whether standard or semi-standard like Boost). All those users banging on it in different ways will help expose lots of bugs. The problem with rapid iteration is that often many new bugs could be inserted -- but I admit this can be mitigated by simply doing something like having separate stable and development branches where the stable branch doesn't change frequently except for bug fixes vs. the dev branch that contains all the new hotness but may be buggier. Again, though, if there was a "Python Boost" that nearly everyone using Python used, this would also accomplish the goal or having something that has tons of users banging on it to expose bugs.
Still, that doesn't mean you shouldn't have a STL of some sort. The purpose of an STL isn't necessarily to have the hottest new features, but to have stuff that's rock-stable, always available, and doesn't need to be rapidly iterated. The C++ std::string class, for example, hasn't really changed much since the 90's, and, frankly, it doesn't need to. Any bugs in it have been worked out long ago and it does a perfect fine job of providing a string type to your code. If Python's STL is broken in some way and not being properly maintained, then either the broken feature needs to be removed from the STL (as it fails the test of being rock-stable) or, dammit, it needs to be fixed and made rock-stable.
Maybe the problem is that Python's STL is simply too far-reaching in functionality to truly be an STL. C++'s provides core functionality like strings, I/O, memory management, containers (lists, vectors, etc.), and so on. If you need anything more exotic (regular expressions, fancy numeric types, threads, etc.), then you usually end up turning to Boost (although some of these Boost features, like threads, are showing up in the newest versions of the C++ standard library as they are now considered core features of modern programs). Granted, Python doesn't necessarily need features like these in its STL as they're built-in or unnecessary to the language itself, but a similar rule of thumb may not be a bad idea. Having an XML parser in the Python standard library is certainly useful, but if it's just going to sit there and rot and only be used by a handful of the myriad of Python apps in the world, maybe it shouldn't be in the STL.
I will say this, though -- ease of use of installing libraries via pip or whatnot is fine if you're writing a serious Python app but even that is a pain in the ass if you simply writing some basic scripts that you want to distribute where the only requirement is that you have a working Python installation. If the test scripts I wrote were only going to be used by me, then yeah, I'd have no problem with installing whatever other random Python package in order to make them work. However, these scripts were meant to be used by my co-workers in addition to myself and I didn't want to make them have to jump through all sorts of hoops just to get those scripts running on their test boxes.
As for distribution of Python software with external requirements, you have a point, but there are many solutions to that.
The best solution is to simply use setuptools! Then instead of giving your co-workers code on a USB stick, they simply do
pip install louscodeDONE!
Oh, you don't want to share your code publicly? Well, you have an internal way to host your code right? Perhaps your git repository? Tell your co-workers to do this:
pip install git://localserver.office/Lou/louscode.git=louscodeNow, if you want to educate your co-workers are bit, like mine are, you can simply include a requirements.txt file in your project. Then it's up to your co-workers to know that they have to do this: Four commands, one of which is cd. Not too shabby if I say so myself. The mkvirtualenv is even optional, if you don't care about isolating the libraries (but you do).
Yet another option is to use your system package manager, like apt! Create a .deb with proper dependencies. Then when your co-workers apt-get install it, it will automatically also install any python-XXX packages that you specified, and all the major ones are usually available.
Now, there are indeed some times that this can screw up. For example, let's say you are doing image processing, and PIL(Pillow is better) is a requirement. Also, you are going to be processing jpegs. If all your co-workers do is pip install pillow, jpeg support will not be there. It will error at runtime when they try to edit a jpeg. This is because they need to have libjpeg-dev already installed via the OS. Most Python packages do not have C-libraries as requirements, but a few do. The ones I encounter most often are usually just wrappers for C libraries in the first place. Examples are the aforementioned PIL, numpy/scipy, DB libraries like MySQL-Python and psycopg2 (postgres), pylibmc (memcached), etc. If all you are doing is sharing scripts with co-workers it is much more likely that the libraries you are using are going to be pure Python, and will not have this issue.
For distribution, you are right in that we do keep it in our source code repository (we don't use git, but the exact tool doesn't really matter in this case), so it's not like I'm handing out USB sticks.
The question is whether all the extra effort to bundle it with the non-standard libraries is worth it. Given how I figured out how to cope with the bug that was causing those script failures early on, it doesn't seem like it would be worth it, at least for now.
http://www.py2exe.org/
As for adding pip to the OS image, maybe you won't have to. easy_install (setuptools) has a very good chance of already being there. pip is really just a better easy_install that allows you do things like use requirements.txt files and install direct from Git repos. You can just easy_install everything. You can even just include setup.py files and do
python setup.pyPS: the dialogues are reminiscent of GEB, if you're into that kind of thing.
Edit: stackoverflow got my back. The answer? Refer to sys/socket.h::send as ::send.
I will start heavy on Java and Python in a couple of weeks.