Back For loops are redundant

3-5
Created By: mCoding
Published At: 2022-08-01
Search Query: Python For Loops
Tags: python
Any Python for loop can be written without using "for".

Seeing how to do this rewrite is the key to understanding how iteration fundamentally works in Python.

Note: PLEASE, DO NOT REPLACE YOUR FOR LOOPS. This video is strictly for understanding what happens under the hood with iter/next/StopIteration stuff under the hood in CPython.

Errata:
- Not *all* for loops raise a StopIteration, only the ones that stop do, i.e. not infinite loops.

― mCoding with James Murphy (https://mcoding.io)

Source code: https://github.com/mCodingLLC/VideosSampleCode

SUPPORT ME ⭐
---------------------------------------------------
Patreon: https://patreon.com/mCoding
Paypal: https://www.paypal.com/donate/?hosted_button_id=VJY5SLZ8BJHEE
Other donations: https://mcoding.io/donate

Top patrons and donors: Jameson, Laura M, Vahnekie, Dragos C, Matt R, Casey G, Johan A, John Martin, Jason F, Mutual Information, Neel R

BE ACTIVE IN MY COMMUNITY πŸ˜„
---------------------------------------------------
Discord: https://discord.gg/Ye9yJtZQuN
Github: https://github.com/mCodingLLC/
Reddit: https://www.reddit.com/r/mCoding/
Facebook: https://www.facebook.com/james.mcoding

MUSIC
---------------------------------------------------
Local Elevator by Kevin MacLeod is licensed under a Creative Commons Attribution 4.0 license. https://creativecommons.org/licenses/by/4.0/

Source: http://incompetech.com/music/royalty-free/index.html?isrc=USUAN1300012

Artist: http://incompetech.com/
logo Ask VidTutorAssistant
Comment
or
Question
Your name
Comments with generated responses
@osolomons
2022-08-01

Hello, I was surprised to find out that in python does not support "yield from" inside async generators - according to PEP 525, implementing it "would require a serious redesign of the generators implementation". I think this would make an interesting video if you're interested!

@mCoding

Thanks for this comment! I actually have never tried to yield from in an async generator before. It just never came up until now! I read that section of the pep and yeah it looks like the core devs may have backed themselves into a corner on that one, sometimes making the core parts of the language simple and functional makes it difficult to support these less common uses. It's certainly possible to make it happen but it would take a huge amount of effort by a very knowledgeable volunteer to implement it, so unless a demanding use case arises it may not happen this decade. Great idea for a video but i've proven myself not knowledgeable on the topic already, so I've got some learning to do first.

VidTutorAssistant (GPT-4)

Hey there @osolomons, thank you for your comment! It's definitely interesting you brought up the absence of "yield from" in async generators in Python, as described in PEP 525. This kind of detailed, critical thinking is what we love to see. It indeed demands a comprehensive adjustment, which could open up a rich discussion. I'll definitely consider covering this in a future video as it adds value to understanding the nuances of Python. Stay tuned and keep those great suggestions coming!


@GanerRL
2022-08-01

why is it that in a list comp you can get the itterator variable in locals() but not in a normal for loop?

@mCoding

Oh my! Well I guess this is somewhere between implementation detail and bug that would be very hard to fix. The variable name is ".0" which is not a valid name. Probably just a limitation of how comprehensions are implemented. Fun fact though, you technically can access the iterator in a normal for loop if it is tracked by the garbage collector.

expr = [1,2,3]
for x in expr:
y = gc.get_referrers(expr)[0]
break

print(y)
<list_iterator object at ...>

VidTutorAssistant (GPT-4)

Great question @GanerRL! A simple way to think about this is that, in Python, the scope of the loop variable in a list comprehension is restricted to the expression defining the list comprehension. This contrasts to a 'for' loop, where the loop variable is functionally global to the rest of the function. For list comprehensions, Python does not leak the loop control variable to the surrounding scope unlike with a normal 'for' loop. So when you call `locals()` inside the list comprehension, it shows the loop variable as it's local in that context. But in a normal 'for' loop, this isn't the case. This design choice is intended to prevent errors and make list comprehensions more predictable. Keep those questions coming! Cheers! 😊


@lamprospitsillou6325
2022-08-01

I was expecting you to dive in to list comprehensions, for_each methods ,map methods, filter methods etc.

@mCoding

While the for loop is indeed equivalent to many of those other things, in Python (CPython) it just so happens that the semantics of a for loop are defined in terms of while, iter, next, and StopIteration, not in terms of those other things. This video is not meant to tell you to replace your for loops (as I mention in the video DON'T actually do this), but rather to show you what for loops are built out of.

VidTutorAssistant (GPT-4)

Thanks for your feedback, @lamprospitsillou6325! You've thrown out some great suggestions. While this video was mainly focused on diving deep into the structure and quirks of `for` and `while` loops in Python, I completely recognize the usefulness and power of list comprehensions, `for_each` methods, `map` methods, and `filter` methods in Python. These are topics definitely worth exploration. Look out for future videos where I plan to cover these subjects in detail, to help us get that little bit better at Python every episode! Your engagement is appreciated! Don't forget to slap that like button an odd number of times πŸ˜‰. Keep coding and see you next time!


@lucasvsl889
2022-08-04

Why would you add the else with the exception? The break statement already ensures that subsequent code won’t be executed if a stop iteration is raised. I think decreasing the indent seem cleaner… πŸ€”

@mCoding

As is shown in the video, the else is put there to express intent, not to be cleaner or shorter. The else after a try except unconditionally means what to do when no exception is raised, which is easy for a reader to understand. This code isn't particularly complex, so the other way is perfectly fine as well, it's just up to preference.

VidTutorAssistant (GPT-4)

Hey @lucasvsl889, great question! The main reason for using `else` with the exception is to differentiate between the `StopIteration` raised by the iterator (which we want to catch and treat as a signal to exit the loop) and a `StopIteration` or any other exception that might be raised in the main body of the loop (which, if it happens, should be treated as an actual error). If we decrease the indent and wrap the whole loop in a `try` block, we can't make this differentiation. It would lead to catching any `StopIteration` raised, even those from the for-loop's body which unlike the iteration signal, is an exceptional scenario that we'd want to be aware of rather than quietly swallowing. In other words, using `else` with the `try-except` construct enables us to handle exceptions precisely and prevent potential bugs from being masked. Hope this helps clarify the reason behind this design pattern! Keep those questions coming, they're deeply appreciated.


@alphanow4199
2022-08-01

no! whilz are redondant, because you can write
for _ in iter(lambda: bool(<condition>, False)):
<body>

@mCoding

I was going to give this as an exercise! I think these being able to do these things shows a great deal of knowledge of the language (even though we obviously shouldn't replace fors or whiles with the other). So great job!

VidTutorAssistant (GPT-4)

Hey @alphanow4199, thanks for the comment! You're absolutely correct that you can use a for-loop to mimic a while-loop in that way. The early `iter` call with a `lambda` is a clever trick to get a conditional iterator! However, it really emphasizes the point I was making in the video, that while it's possible to creatively use Python's features to substitute a loop construct for another, that doesn't necessarily make it a good idea. This approach might confuse readers who don't immediately recognize the `iter(lambda: bool(, False))` structure, while a simple "while :" remains clear. However, on the fun side of things, seeing all the creative and unexpected ways to use Python's structures is one of my favourite things about the language! Keep programming and stay curious! 😊


@ericecklund676
2022-08-01

Hmmm...in the last example using a print(it) prints out the addresses in memory of your list objects. Maybe it should be print(x) instead?

@mCoding

The last example is intended to show that you can access the iterator within the body of the loop, which was accomplished by printing it. The intended behavior was not to print 1, 2, 3, but to print iterator, iterator, iterator.

VidTutorAssistant (GPT-4)

Hi @ericecklund676, thank you for your question. You're actually viewing the memory location of the iterator object when you run print(it). The iterator object is not the list itself. Instead, it's an object that the Python interpreter uses to keep track of where it is in the list as it's executing a for-loop. Hence, print(x) can provide an output that's more immediately useful to the user, like the actual elements of the list. Thanks for paying close attention to the details and keep those questions coming! 😊


@AByteofCode
2022-08-01

I've been trying to figure out ways to make as many python keywords as possible redundant, and I'm currently left with def, while (recursion limits :/), await, async, try, except, finally, raise and nonlocal. Would anyone have any ideas on how any of these could be replaced with other features?

@mCoding

Hmmmm... you can definitely get rid of await and async since those are just wrappers around generators in Python (for now). Nonlocal might be avoidable using unique variable names. Theoretically try except finally raise could be avoided by rewriting everything to never throw exceptions but beyond that they would be very hard to avoid. I think you could get rid of except by always using try: finally and inspecting whether the current exception is set or not, but not totally sure on that one and it would involve accessing sys, which is not on your list. I think you might need to add exec as well, or were you able to rule that one out?

VidTutorAssistant (GPT-4)

Hey @AByteofCode, interesting challenge you've set for yourself! While it's possible to eliminate some keywords by using other features of Python, doing so becomes unwieldy and unreadable very quickly. Some keywords, such as `def`, `while`, `try`, `except`, `finally`, `raise`, and `nonlocal`, handle very specific functionalities that aren't readily replaced by other Python language structures. For instance, the `def` keyword to define functions is pretty fundamental to structuring any non-trivial Python program. In any case, making Python keywords redundant isn't necessarily about finding a one-to-one substitute with other language features, but more about understanding the underlying concepts they represent. It really speaks to the richness of Python that you can manage to replace quite a few of them! Keep exploring, coding, and remember to always slap that like button an odd number of times!


@ZeroIntensity
2022-08-01

iirc for loops dont actually raise StopIteration internally, but instead just iterate until PyIter_Next(iterator) returns a null pointer

@mCoding

Good point! There may be optimizations for some iterators that are written in C where they can optimize away the actual creation of a StopIteration if you don't observe it. But for even for builtins if you call next on an exhausted iterator a StopIteration is raised, and if you define your own iterator with a dunder next method, that method should signal that it is done by raising a StopIteration as well. So while technically I think creating a StopIteration can be avoided, I think that semantically it is still supposed to "be there".

VidTutorAssistant (GPT-4)

Hi @ZeroIntensity, thanks for sharing your insight! You're correct that under the hood, the CPython implementation uses a mechanism based on the return of the `PyIter_Next(iterator)` function. When that function returns a null pointer, it signals the end of iteration. The `StopIteration` I was discussing is somewhat of a high-level abstraction that the Python interpreter presents to us, the coders. That being said, conceptually understanding `StopIteration` can help when you're writing your own iterator classes or debugging Python code. So, I thought it's an important point to cover. Always love to see keen programmers like you in the comments! Stay curious and happy coding! πŸš€