Mailing List Archive

on perhaps unloading modules?
(*) Introduction

By the way, I'm aware that what I'm doing here is totally unsafe and I
could get my system destroyed. I'm not planning on using this --- thank
you for your concern. I'm just interested in understanding more about
modules.

(*) The problem

I got myself into a mess with loading modules at runtime. The reason
I'm loading modules at runtime is because I'm writing a script to grade
students' work. (Each student test is a file.py.)

Here's the phenomenon. Notice how student m0 (who really scored a zero)
first gets his grade right, but if I invoke it again, then it gets 50.0.

>>> grade_student(m0)
{'grade': 0.0, ...}
>>> grade_student(m0)
{'grade': 50.0, ...}

(*) Where's the problem?

The variable m0 is a module and modules in Python are effectively
singletons --- that is, they are meant to be loaded only once, no matter
how many times you ask the system to. That's my understanding.

(*) How did I get into this mess?

When m0 is graded first, that is the first student's module ever loaded,
so everything turns out correct. This module m0 didn't do any work ---
didn't write any procedures, so it gets grade = 0 ---, so when I check
the first procedure, it doesn't exist --- zero on question 1. However,
question 2 depends on question 1. So I use the test's key (which I
wrote, which is perfectly correct) and I augment the student's module
with the key's procedures that are prerequisites to question 2 and then
I test question 2. How do I do that?

I do m.question1 = key.question1

where ``key.question1'' is a correct procedure for getting all points of
question1. (That's kind to the student: I'm allowing them to get a zero
on question 1 while perhaps getting question 2 right.) However, once I
augment the student's code, I can' t find a way to properly restore it
to the original --- so on a second execution, m0 gets many more points.

That's not the whole problem. For reasons I don't understand, new
modules I load --- that is, different students --- get mixed with these
modifications in m0 that I made at some point in my code.

Of course, you want to see the code. I need to work on producing a
small example. Perhaps I will even answer my own question when I do.

For now, let me just ignite your imagination. Feel free to ignore all
of this and wait for a nice example of the problem.

(*) The code

How do I load a student's file?

--8<---------------cut here---------------start------------->8---
from importlib import *
def get_student_module(fname):
# Here fname := p1_hello.py. But we need to remove the extension.
# Let's call it basename then.
mod_name = basename(fname)
try:
student = import_module(mod_name)
except Exception as e:
return False, str(e)
return True, student
--8<---------------cut here---------------end--------------->8---

Now let d be a path to a directory. How do use I this procedure?

--8<---------------cut here---------------start------------->8---
for f in get_all_tests(d):
okay, student = get_student_module(f)
report = grade_student(student)
[...]
--8<---------------cut here---------------end--------------->8---

What does grade_student(student_module) do? It passes student_module to
procedures to check every little thing the test requires. Let's take a
look at question4().

Question 4 requires a previous procedure called procedure_x. So I
overwrite the student's module with the key's procedure.

def question4(m):
# We first provide students with all the prerequisites of the
# question, so they don't necessarily get this one wrong by getting
# prerequisites wrong.
m.procedure_x = key.procedure_x

# Now I make all my verifications about, say, m.procedure_y and I
# compute an integer called losses, which I return.

return losses

This strategy must be so unwise that it totally breaks my naïve dream of
automatic grading, showing how unfit I am for teaching students how to
program. Such is life.

(*) If it were easy to unload modules...

I could just unload it and load it again. That should restore the
student's module back to its original source code.
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
Hope Rouselle <hrouselle@jevedi.com> writes:

[...]

> Of course, you want to see the code. I need to work on producing a
> small example. Perhaps I will even answer my own question when I do.

[...]

Here's a small-enough case. We have two students here. One is called
student.py and the other is called other.py. They both get question 1
wrong, but they --- by definition --- get question 2 right. Each
question is worth 10 points, so they both should get losses = 10.

(*) Student student.py

--8<---------------cut here---------------start------------->8---
def question1(t): # right answer is t[2]
return t[1] # lack of attention, wrong answer
--8<---------------cut here---------------end--------------->8---

(*) Student other.py

--8<---------------cut here---------------start------------->8---
def question1(t): # right answer is t[2]
return t[0] # also lack of attention, wrong answer
--8<---------------cut here---------------end--------------->8---

(*) Grading

All is good on first run.

Python 3.5.2 [...] on win32
[...]
>>> reproducible_problem()
student.py, total losses 10
other.py, total losses 10

The the problem:

>>> reproducible_problem()
student.py, total losses 0
other.py, total losses 0

They lose nothing because both modules are now permanently modified.

(*) The code of grading.py

--8<---------------cut here---------------start------------->8---
# -*- mode: python; python-indent-offset: 2 -*-
def key_question1(t):
# Pretty simple. Student must just return index 2 of a tuple.
return t[2]

def reproducible_problem(): # grade all students
okay, m = get_student_module("student.py")
r = grade_student(m)
print("student.py, total losses", r) # should be 10
okay, m = get_student_module("other.py")
r = grade_student(m)
print("other.py, total losses", r) # should be 10

def grade_student(m): # grades a single student
losses = question1_verifier(m)
losses += question2_verifier(m)
return losses

def question1_verifier(m):
losses = 0
if m.question1( (0, 1, 2, 3) ) != 2: # wrong answer
losses = 10
return losses

def question2_verifier(m):
m.question1 = key_question1
# To grade question 2, we overwrite the student's module by giving
# it the key_question1 procedure. This way we are able to let the
# student get question 2 even if s/he got question 1 incorrect.
losses = 0
return losses

def get_student_module(fname):
from importlib import import_module
mod_name = basename(fname)
try:
student = import_module(mod_name)
except Exception as e:
return False, str(e)
return True, student

def basename(fname): # drop the the .py extension
return "".join(fname.split(".")[ : -1])
--8<---------------cut here---------------end--------------->8---
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
Hope Rouselle <hrouselle@jevedi.com> writes:

> Hope Rouselle <hrouselle@jevedi.com> writes:
>
> [...]
>
>> Of course, you want to see the code. I need to work on producing a
>> small example. Perhaps I will even answer my own question when I do.
>
> [...]
>
> Here's a small-enough case. We have two students here. One is called
> student.py and the other is called other.py. They both get question 1
> wrong, but they --- by definition --- get question 2 right. Each
> question is worth 10 points, so they both should get losses = 10.
>
> (*) Student student.py
>
> def question1(t): # right answer is t[2]
> return t[1] # lack of attention, wrong answer
>
>
> (*) Student other.py
>
> def question1(t): # right answer is t[2]
> return t[0] # also lack of attention, wrong answer
>
>
> (*) Grading
>
> All is good on first run.
>
> Python 3.5.2 [...] on win32
> [...]
>>>> reproducible_problem()
> student.py, total losses 10
> other.py, total losses 10
>
> The the problem:
>
>>>> reproducible_problem()
> student.py, total losses 0
> other.py, total losses 0
>
> They lose nothing because both modules are now permanently modified.
>
> (*) The code of grading.py
>
> # -*- mode: python; python-indent-offset: 2 -*-
> def key_question1(t):
> # Pretty simple. Student must just return index 2 of a tuple.
> return t[2]
>
> def reproducible_problem(): # grade all students
> okay, m = get_student_module("student.py")
> r = grade_student(m)
> print("student.py, total losses", r) # should be 10
> okay, m = get_student_module("other.py")
> r = grade_student(m)
> print("other.py, total losses", r) # should be 10
>
> def grade_student(m): # grades a single student
> losses = question1_verifier(m)
> losses += question2_verifier(m)
> return losses
>
> def question1_verifier(m):
> losses = 0
> if m.question1( (0, 1, 2, 3) ) != 2: # wrong answer
> losses = 10
> return losses
>
> def question2_verifier(m):
> m.question1 = key_question1
> # To grade question 2, we overwrite the student's module by giving
> # it the key_question1 procedure. This way we are able to let the
> # student get question 2 even if s/he got question 1 incorrect.
> losses = 0
> return losses

My solution is to painfully save each original procedure from student in
a variable, replace them with the key's, then restore them at the end.
I couldn't overwrite m.__dict__ because it is readonly. (Didn't find a
neater's way out.)

def question2_verifier(m):
question1_original = m.question1
m.question1 = key_question1
# apply verifications...
m.question1 = question1_original
return losses
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
On 16/08/21 1:50 am, Hope Rouselle wrote:
> By the way, I'm aware that what I'm doing here is totally unsafe and I
> could get my system destroyed. I'm not planning on using this --- thank
> you for your concern. I'm just interested in understanding more about
> modules.

Okay, I'll assume all the security issues have been taken are of, e.g.
by running all of this in a virtual machine...

> Notice how student m0 (who really scored a zero)
> first gets his grade right, but if I invoke it again, then it gets 50.0.

The best way to do this would be to run each student's file in
a separate process, so you know you're getting a completely fresh
start each time.

The second best way would be to not use import_module, but to
exec() the student's code. That way you don't create an entry in
sys.modules and don't have to worry about somehow unloading the
module.

Something like

code = read_student_file(student_name)
env = {} # A dict to hold the student's module-level definitions
exec(code, env)
grade_question1(env)
env['procedure_x'] = key.procedure_x
grade_question2(env)
...etc...

> That's not the whole problem. For reasons I don't understand, new
> modules I load --- that is, different students --- get mixed with these
> modifications in m0 that I made at some point in my code.

I would have to see a specific example of that. One thing to keep
in mind is that if key.procedure_x modifies any globals in the
key module, it will still modify globals in the key module -- not
the student's module -- after being transplanted there.

More generally, there are countless ways that a student's code
could modify something outside of its own module and affect the
behaviour of other student's code. This is why it would be
vastly preferable to run each test in a fresh process.

> (*) If it were easy to unload modules...

It's sometimes possible to unload and reimport a module, but
only if the module's effects are completely self-contained.
That depends not only on what the module itself does, but
what other modules do with it. If any other module has imported
it, that module will still contain references to the old
module; if there are instances of a class defined in it still
existing, they will still be instances of the old version of
the class; etc.

99.999% of the time it's easier to just start again with a
fresh Python process.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
Greg Ewing <greg.ewing@canterbury.ac.nz> writes:

> On 16/08/21 1:50 am, Hope Rouselle wrote:
>> By the way, I'm aware that what I'm doing here is totally unsafe and I
>> could get my system destroyed. I'm not planning on using this --- thank
>> you for your concern. I'm just interested in understanding more about
>> modules.
>
> Okay, I'll assume all the security issues have been taken are of, e.g.
> by running all of this in a virtual machine...

Oh, good idea. :-D

>> Notice how student m0 (who really scored a zero)
>> first gets his grade right, but if I invoke it again, then it gets 50.0.
>
> The best way to do this would be to run each student's file in
> a separate process, so you know you're getting a completely fresh
> start each time.

Yes, that would be much better indeed, but I'd be left with IPC
mechanisms of exchanging data. My first idea is to just print out a
JSON of the final report to be read by the grader-process. (That is
essentially what I'm already doing, except that I don't need to
serialize things in a string.)

> The second best way would be to not use import_module, but to
> exec() the student's code. That way you don't create an entry in
> sys.modules and don't have to worry about somehow unloading the
> module.
>
> Something like
>
> code = read_student_file(student_name)
> env = {} # A dict to hold the student's module-level definitions
> exec(code, env)
> grade_question1(env)
> env['procedure_x'] = key.procedure_x
> grade_question2(env)
> ...etc...

That seems a lot easier to implement.

>> That's not the whole problem. For reasons I don't understand, new
>> modules I load --- that is, different students --- get mixed with these
>> modifications in m0 that I made at some point in my code.
>
> I would have to see a specific example of that.

I'm sorry. I don't think I was right in that observation. I tried to
produce one such small program to post here and failed. (Let's forget
about that.)

> One thing to keep in mind is that if key.procedure_x modifies any
> globals in the key module, it will still modify globals in the key
> module -- not the student's module -- after being transplanted there.
>
> More generally, there are countless ways that a student's code
> could modify something outside of its own module and affect the
> behaviour of other student's code. This is why it would be
> vastly preferable to run each test in a fresh process.

I'm totally convinced. I'll do everything in a fresh process. Having
guarantees makes the job much more pleasant to do.

>> (*) If it were easy to unload modules...
>
> It's sometimes possible to unload and reimport a module, but
> only if the module's effects are completely self-contained.
> That depends not only on what the module itself does, but
> what other modules do with it. If any other module has imported
> it, that module will still contain references to the old
> module; if there are instances of a class defined in it still
> existing, they will still be instances of the old version of
> the class; etc.
>
> 99.999% of the time it's easier to just start again with a
> fresh Python process.

I'm totally convinced. Thank you so much!
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
On Tue, Aug 17, 2021 at 4:02 AM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
> The second best way would be to not use import_module, but to
> exec() the student's code. That way you don't create an entry in
> sys.modules and don't have to worry about somehow unloading the
> module.

I would agree with this. If you need to mess around with modules and
you don't want them to be cached, avoid the normal "import" mechanism,
and just exec yourself a module's worth of code.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
This may not answer your question but it may provide an alternative
solution.

I had the same challenge that you an year ago so may be my solution will
work for you too.

Imagine that you have a Markdown file that *documents* the expected
results.

--8<---------------cut here---------------start------------->8---
This is the final exam, good luck!

First I'm going to load your code (the student's code):

```python
>>> import student
```

Let's see if you programmed correctly a sort algorithm

```python
>>> data = [3, 2, 1, 3, 1, 9]
>>> student.sort_numbers(data)
[1, 1, 2, 3, 3, 9]
```

Let's now if you can choose the correct answer:

```python
>>> t = ["foo", "bar", "baz"]
>>> student.question1(t)
"baz"
```
--8<---------------cut here---------------end--------------->8---

Now you can run the snippets of code with:

byexample -l python the_markdown_file.md

What byexample does is to run the Python code, capture the output and
compare it with the expected result.

In the above example "student.sort_numbers" must return the list sorted.
That output is compared by byexample with the list written below.

Advantages? Each byexample run is independent of the other and the
snippet of codes are executed in a separated Python process. byexample
takes care of the IPC.

I don't know the details of your questions so I'm not sure if byexample
will be the tool for you. In my case I evaluate my students giving them
the Markdown and asking them to code the functions so they return the
expected values.

Depending of how many students you have you may considere to complement
this with INGInious. It is designed to run students' assignments
assuming nothing on the untrusted code.

Links:

https://byexamples.github.io/byexample/
https://docs.inginious.org/en/v0.7/


On Sun, Aug 15, 2021 at 12:09:58PM -0300, Hope Rouselle wrote:
>Hope Rouselle <hrouselle@jevedi.com> writes:
>
>[...]
>
>> Of course, you want to see the code. I need to work on producing a
>> small example. Perhaps I will even answer my own question when I do.
>
>[...]
>
>Here's a small-enough case. We have two students here. One is called
>student.py and the other is called other.py. They both get question 1
>wrong, but they --- by definition --- get question 2 right. Each
>question is worth 10 points, so they both should get losses = 10.
>
>(*) Student student.py
>
>--8<---------------cut here---------------start------------->8---
>def question1(t): # right answer is t[2]
> return t[1] # lack of attention, wrong answer
>--8<---------------cut here---------------end--------------->8---
>
>(*) Student other.py
>
>--8<---------------cut here---------------start------------->8---
>def question1(t): # right answer is t[2]
> return t[0] # also lack of attention, wrong answer
>--8<---------------cut here---------------end--------------->8---
>
>(*) Grading
>
>All is good on first run.
>
>Python 3.5.2 [...] on win32
>[...]
>>>> reproducible_problem()
>student.py, total losses 10
>other.py, total losses 10
>
>The the problem:
>
>>>> reproducible_problem()
>student.py, total losses 0
>other.py, total losses 0
>
>They lose nothing because both modules are now permanently modified.
>
>(*) The code of grading.py
>
>--8<---------------cut here---------------start------------->8---
># -*- mode: python; python-indent-offset: 2 -*-
>def key_question1(t):
> # Pretty simple. Student must just return index 2 of a tuple.
> return t[2]
>
>def reproducible_problem(): # grade all students
> okay, m = get_student_module("student.py")
> r = grade_student(m)
> print("student.py, total losses", r) # should be 10
> okay, m = get_student_module("other.py")
> r = grade_student(m)
> print("other.py, total losses", r) # should be 10
>
>def grade_student(m): # grades a single student
> losses = question1_verifier(m)
> losses += question2_verifier(m)
> return losses
>
>def question1_verifier(m):
> losses = 0
> if m.question1( (0, 1, 2, 3) ) != 2: # wrong answer
> losses = 10
> return losses
>
>def question2_verifier(m):
> m.question1 = key_question1
> # To grade question 2, we overwrite the student's module by giving
> # it the key_question1 procedure. This way we are able to let the
> # student get question 2 even if s/he got question 1 incorrect.
> losses = 0
> return losses
>
>def get_student_module(fname):
> from importlib import import_module
> mod_name = basename(fname)
> try:
> student = import_module(mod_name)
> except Exception as e:
> return False, str(e)
> return True, student
>
>def basename(fname): # drop the the .py extension
> return "".join(fname.split(".")[ : -1])
>--8<---------------cut here---------------end--------------->8---
>--
>https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
Chris Angelico <rosuav@gmail.com> writes:

> On Tue, Aug 17, 2021 at 4:02 AM Greg Ewing
> <greg.ewing@canterbury.ac.nz> wrote:
>> The second best way would be to not use import_module, but to
>> exec() the student's code. That way you don't create an entry in
>> sys.modules and don't have to worry about somehow unloading the
>> module.
>
> I would agree with this. If you need to mess around with modules and
> you don't want them to be cached, avoid the normal "import" mechanism,
> and just exec yourself a module's worth of code.

Sounds like a plan. Busy, haven't been able to try it out. But I will.
Soon. Thank you!
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
Martin Di Paola <martinp.dipaola@gmail.com> writes:

> This may not answer your question but it may provide an alternative
> solution.
>
> I had the same challenge that you an year ago so may be my solution will
> work for you too.
>
> Imagine that you have a Markdown file that *documents* the expected
> results.
>
> This is the final exam, good luck!
>
> First I'm going to load your code (the student's code):
>
> ```python
>>>> import student
> ```
>
> Let's see if you programmed correctly a sort algorithm
>
> ```python
>>>> data = [3, 2, 1, 3, 1, 9]
>>>> student.sort_numbers(data)
> [1, 1, 2, 3, 3, 9]
> ```
>
> Let's now if you can choose the correct answer:
>
> ```python
>>>> t = ["foo", "bar", "baz"]
>>>> student.question1(t)
> "baz"
> ```
>
> Now you can run the snippets of code with:
>
> byexample -l python the_markdown_file.md
>
> What byexample does is to run the Python code, capture the output and
> compare it with the expected result.
>
> In the above example "student.sort_numbers" must return the list
> sorted.
> That output is compared by byexample with the list written below.
>
> Advantages? Each byexample run is independent of the other and the
> snippet of codes are executed in a separated Python process. byexample
> takes care of the IPC.
>
> I don't know the details of your questions so I'm not sure if byexample
> will be the tool for you. In my case I evaluate my students giving them
> the Markdown and asking them to code the functions so they return the
> expected values.

Currently procedures in one question are used in another question.
Nevertheless, perhaps I could (in other tests) design something
different. Although, to be honest, I would rather not have to use
something like Markdown because that means more syntax for students.

> Depending of how many students you have you may considere to
> complement this with INGInious. It is designed to run students'
> assignments assuming nothing on the untrusted code.
>
> Links:
>
> https://byexamples.github.io/byexample/
> https://docs.inginious.org/en/v0.7/

INGInious looks pretty interesting. Thank you!
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
Chris Angelico <rosuav@gmail.com> writes:

> On Tue, Aug 17, 2021 at 4:02 AM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
>> The second best way would be to not use import_module, but to
>> exec() the student's code. That way you don't create an entry in
>> sys.modules and don't have to worry about somehow unloading the
>> module.
>
> I would agree with this. If you need to mess around with modules and
> you don't want them to be cached, avoid the normal "import" mechanism,
> and just exec yourself a module's worth of code.

This is looks very interesting.

--8<---------------cut here---------------start------------->8---
Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:01:18) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> >>> code
<function code at 0x009F5660>
>>> code()
'def p():\n import math\n return math.e\n'
>>> p
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'p' is not defined
>>> exec(code())
>>> p
<function p at 0x0113F5D0>
>>> p()
2.718281828459045
--8<---------------cut here---------------end--------------->8---

This is also totally unsafe, right? But at least it won't give me those
loading problems I had before. But I think I'm gonna go with Greg
Ewing's first option --- running it all on a separate process, computing
that report I was computing, then printing it out to the stdout and
reading it from a controller process to know the student's results.

This way I'm one step closer to safety --- I could chroot it and give it
a limited period of time for execution. I suppose I could also
customize Python's environment to only allow the modules I'd like to
allow as well as the builtins I'd like to allow. (That would be
interesting as well.)

So, alright, let me learn how to spawn a process. Thanks very much.
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
On 21/08/21 6:15 am, Hope Rouselle wrote:
>>>> code()
> 'def p():\n import math\n return math.e\n'
>>>> exec(code())
>>>> p
> <function p at 0x0113F5D0>
>>>> p()
> 2.718281828459045

Note that this pollutes the globals of the module that you're calling
exec() from. For better isolation you can pass in an explicit globals
dict:

g = {}
exec(code(), g)
g['p']()

--
Greg

--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
Greg Ewing <greg.ewing@canterbury.ac.nz> writes:

> On 21/08/21 6:15 am, Hope Rouselle wrote:
>>>>> code()
>> 'def p():\n import math\n return math.e\n'
>>>>> exec(code())
>>>>> p
>> <function p at 0x0113F5D0>
>>>>> p()
>> 2.718281828459045
>
> Note that this pollutes the globals of the module that you're calling
> exec() from. For better isolation you can pass in an explicit globals
> dict:
>
> g = {}
> exec(code(), g)
> g['p']()

Oh! Now I understand how to use it! That's in fact what I was looking
for. I noticed it was polluting my environment and was thinking --- hm,
that's no good. Thank you.

So I believe I understand how to pollute their environment too. Say I
have a procedure called external that I'd like to make available to
them. It seems this is what I need to do.

--8<---------------cut here---------------start------------->8---
def external():
return "external"

def run():
s = """
def s(*args):
import math
return external(), math.e, args
"""
g = {}
exec(s, g)
g["external"] = external
return g
--8<---------------cut here---------------end--------------->8---

>>> student["s"](1, 2)
('external', 2.718281828459045, (1, 2))

That's good. So I can restrict their environment too, by removing some
built-ins and so on. (I wish I could restrict their syntax too, though,
but I fear that's not possible. For instance, it would be very useful
if I could remove loops. If the course doesn't let them use them, it is
silly to have to ask them kindly not to use them --- please be patient
with this poorly designed course. In reality there is a whole
department learning to run a course and there are many students helping
this department get a passing grade into how to do it.) :-D

Anyhow, your ideas have improved the outlook of this grader quite a lot.
Thank you!
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
On 21/08/21 1:36 pm, Hope Rouselle wrote:
> I wish I could restrict their syntax too, though,
> but I fear that's not possible. For instance, it would be very useful
> if I could remove loops.

Actually you could, using ast.parse to get an AST and then walk
it looking for things you don't want to allow.

You could also play around with giving them a custom set of
builtins and restricting what they can import. Just be aware
that such things are not foolproof, and a sufficiently smart
student could find ways around them. (Although if they know
that much they probably deserve to pass the course anyway!)

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
Greg Ewing <greg.ewing@canterbury.ac.nz> writes:

> On 21/08/21 1:36 pm, Hope Rouselle wrote:
>> I wish I could restrict their syntax too, though, but I fear that's
>> not possible. For instance, it would be very useful if I could
>> remove loops.
>
> Actually you could, using ast.parse to get an AST and then walk
> it looking for things you don't want to allow.

Very interesting! Thanks very much. That would let me block them,
though the ideal would be a certain python.exe binary that simply blows
a helpful syntax error when they use something the course doesn't allow.
But surely the course could provide students with a certain module or
procedure which would validate their work. (Don't turn in unless you
pass these verifications.)

> You could also play around with giving them a custom set of
> builtins and restricting what they can import. Just be aware
> that such things are not foolproof, and a sufficiently smart
> student could find ways around them. (Although if they know
> that much they probably deserve to pass the course anyway!)

So true! If they can get around such verifications, they should present
their work at an extra-curricular sessions.
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
On Sun, Aug 22, 2021 at 4:37 AM Hope Rouselle <hrouselle@jevedi.com> wrote:
>
> Greg Ewing <greg.ewing@canterbury.ac.nz> writes:
>
> > On 21/08/21 1:36 pm, Hope Rouselle wrote:
> >> I wish I could restrict their syntax too, though, but I fear that's
> >> not possible. For instance, it would be very useful if I could
> >> remove loops.
> >
> > Actually you could, using ast.parse to get an AST and then walk
> > it looking for things you don't want to allow.
>
> Very interesting! Thanks very much. That would let me block them,
> though the ideal would be a certain python.exe binary that simply blows
> a helpful syntax error when they use something the course doesn't allow.
> But surely the course could provide students with a certain module or
> procedure which would validate their work. (Don't turn in unless you
> pass these verifications.)
>
> > You could also play around with giving them a custom set of
> > builtins and restricting what they can import. Just be aware
> > that such things are not foolproof, and a sufficiently smart
> > student could find ways around them. (Although if they know
> > that much they probably deserve to pass the course anyway!)
>
> So true! If they can get around such verifications, they should present
> their work at an extra-curricular sessions.

Agreed... if they do it knowingly. On the other hand, if they just
turn in code copied blindly from Stack Overflow...

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
Chris Angelico <rosuav@gmail.com> writes:

> On Sun, Aug 22, 2021 at 4:37 AM Hope Rouselle <hrouselle@jevedi.com> wrote:
>>
>> Greg Ewing <greg.ewing@canterbury.ac.nz> writes:
>>
>> > On 21/08/21 1:36 pm, Hope Rouselle wrote:
>> >> I wish I could restrict their syntax too, though, but I fear that's
>> >> not possible. For instance, it would be very useful if I could
>> >> remove loops.
>> >
>> > Actually you could, using ast.parse to get an AST and then walk
>> > it looking for things you don't want to allow.
>>
>> Very interesting! Thanks very much. That would let me block them,
>> though the ideal would be a certain python.exe binary that simply blows
>> a helpful syntax error when they use something the course doesn't allow.
>> But surely the course could provide students with a certain module or
>> procedure which would validate their work. (Don't turn in unless you
>> pass these verifications.)
>>
>> > You could also play around with giving them a custom set of
>> > builtins and restricting what they can import. Just be aware
>> > that such things are not foolproof, and a sufficiently smart
>> > student could find ways around them. (Although if they know
>> > that much they probably deserve to pass the course anyway!)
>>
>> So true! If they can get around such verifications, they should present
>> their work at an extra-curricular sessions.
>
> Agreed... if they do it knowingly. On the other hand, if they just
> turn in code copied blindly from Stack Overflow...

When a student does something impressive, we should recognize it.
(Don't we do this with ourselves? That's why we should. Not because
it's a strategy to sort of encourage the behavior, but because it is our
culture.) If if turns out they're clueless about their own feat, we
probably would notice by recognizing it in the first time. Being so
cool, they could present it to everyone. (This is not a strategy to
sort of catch plagiarism, but rather a sober modus operandi that seems
to work well.) Copying blindly from somewhere could be the event that
triggers a deep interest.

One of the things that attracted me to computers was the keyboard. Not
sure if it was the first thing. But it's pretty. One time I saw
Microsoft Windows and someone let me play on Paint. I loved the Bézier
curves and some of the colors. Then came the Internet. Wow! The
Internet brought me UNIX and I began to see computer code. I was
impressed at how someone could just understand such things. I wanted to
write some PHP precisely because it looked so much more cryptic than
Allaire ColdFusion. Then C looked even more cryptic, so I fell in love
with C.

Maybe a little copy from wherever could be the spark for bigger things.
Education is a fantastically difficult thing and making mistakes seems
so involved with it that it's perhaps undetachable from it. Of course,
we should stop ourselves from making irreversible mistakes such as
falling from a precipice.
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
Hope Rouselle <hrouselle@jevedi.com> writes:

> Chris Angelico <rosuav@gmail.com> writes:
>
>> On Tue, Aug 17, 2021 at 4:02 AM Greg Ewing
>> <greg.ewing@canterbury.ac.nz> wrote:
>>> The second best way would be to not use import_module, but to
>>> exec() the student's code. That way you don't create an entry in
>>> sys.modules and don't have to worry about somehow unloading the
>>> module.
>>
>> I would agree with this. If you need to mess around with modules and
>> you don't want them to be cached, avoid the normal "import" mechanism,
>> and just exec yourself a module's worth of code.
>
> Sounds like a plan. Busy, haven't been able to try it out. But I will.
> Soon. Thank you!

Just to close off this thread, let me share a bit of what I wrote. The
result is a lot better. Thanks for all the help!

I exec the student's code into a dictionary g.

--8<---------------cut here---------------start------------->8---
def fs_read(fname):
with open(fname, "r") as f:
return f.read()

def get_student_module_exec(fname):
g = {}
try:
student_code = fs_read(fname)
student = exec(student_code, g)
except Exception as e:
return False, str(e)
return True, g

def get_student_module(fname):
return get_student_module_exec(fname)
--8<---------------cut here---------------end--------------->8---

And now write the test's key as if I were a student and named my test as
"test_key.py".

--8<---------------cut here---------------start------------->8---
def get_key():
okay, k = get_student_module("test_key.py")
if not okay:
# Stop everything.
...
return g
--8<---------------cut here---------------end--------------->8---

The procedure for grading a question consumes the student's code as a
dictionary /s/, grabs the key as /k/ and checks whether the procedures
are the same. So, suppose I want to check whether a certain function
/fn/ written in the student's dictionary-code /s/ matches the key's.
Then I invoke check_student_procedure(k, s, fn).

--8<---------------cut here---------------start------------->8---
def check_student_procedure(k, s, fn, args = [], wrap = identity):
return check_functions_equal(g[fn], s.get(fn, None), args, wrap)
--8<---------------cut here---------------end--------------->8---

For completeness, here's check_functions_equal.

--8<---------------cut here---------------start------------->8---
def check_functions_equal(fn_original, fn_candidate, args = [], wrap = identity):
flag, e = is_function_executable(fn_candidate, args)
if not flag:
return False, "runtime", e
# run original and student's code, then compare them
answer_correct = fn_original(*args)
answer_student = wrap(fn_candidate(*args))
if answer_correct != answer_student:
return False, None, str(answer_student)
return True, None, None

def identity(x):
return x
--8<---------------cut here---------------end--------------->8---

To explain my extra complication there: sometimes I'm permissive with
student's answers. Suppose a question requires a float as an answer but
in some cases the answer is a whole number --- such as 1.0. If the
student ends up producing an int, the student gets that case right: I
wrap the student's answer in a float() and the check turns out
successful.

I probably don't need to check whether a procedure is executable first,
but I decided to break the procedure into two such steps.

--8<---------------cut here---------------start------------->8---
def is_function_executable(f, args = []):
try:
f(*args)
except Exception as e:
return False, str(e)
return True, None
--8<---------------cut here---------------end--------------->8---
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
On Sat, 21 Aug 2021 17:15:14 -0300, Hope Rouselle <hrouselle@jevedi.com>
declaimed the following:

>write some PHP precisely because it looked so much more cryptic than
>Allaire ColdFusion. Then C looked even more cryptic, so I fell in love
>with C.
>
Try APL then...

(I suspect this will get garbaged in processing...)

4 5 ? 20 ? 52

(it did... the first ? [in my display] is Greek lower case rho, the second
question mark is... a question mark).

4 5 $RHO 20 ? 52

Generate 20 random numbers in the range 1..52, no duplicates, reshape
the vector into a 4x5 matrix.

That just "dealt" four poker hands (needs logic to translate integer
1..52 into suit/rank).


--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed@ix.netcom.com http://wlfraed.microdiversity.freeddns.org/

--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
On 2021-08-22 17:18, Dennis Lee Bieber wrote:
> On Sat, 21 Aug 2021 17:15:14 -0300, Hope Rouselle <hrouselle@jevedi.com>
> declaimed the following:
>
>>write some PHP precisely because it looked so much more cryptic than
>>Allaire ColdFusion. Then C looked even more cryptic, so I fell in love
>>with C.
>>
> Try APL then...
>
> (I suspect this will get garbaged in processing...)
>
> 4 5 ? 20 ? 52
>
4 5 ? 20 ? 52

> (it did... the first ? [in my display] is Greek lower case rho, the second
> question mark is... a question mark).
>
The headers say that the encoding is ASCII.

> 4 5 $RHO 20 ? 52
>
> Generate 20 random numbers in the range 1..52, no duplicates, reshape
> the vector into a 4x5 matrix.
>
> That just "dealt" four poker hands (needs logic to translate integer
> 1..52 into suit/rank).
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
Dennis Lee Bieber <wlfraed@ix.netcom.com> writes:

> On Sat, 21 Aug 2021 17:15:14 -0300, Hope Rouselle <hrouselle@jevedi.com>
> declaimed the following:
>
>>write some PHP precisely because it looked so much more cryptic than
>>Allaire ColdFusion. Then C looked even more cryptic, so I fell in love
>>with C.
>>
> Try APL then...

Lol. If the code below is APL, then thanks, but no, thanks. :-) We
change over the years. These days I'm more or less obsessed with
clarity. For instance, I'm a lot into literate programming Knuth-style.

I have a certain distaste for syntax too. For instance, I would much
rather write and read ``first(ls)'' than ``ls[0]''. I can't see too
much advantage in using more syntax for something that procedures could
do. But, of course, in certain contexts it might be better to write
ls[0] or something like that. In these cases, the writer could then
make the case and then define the new syntax, so I do think that writers
should have the power to change syntax.

I'm well aware of the double-edged sword this is. (Incidentally,
yesterday I was happily reading a Python programmer talking about [1]
how Dijkstra got it right for suggesting we shouldn't be using goto.
Knuth had the opinion that it'd be a shame to remove goto, even though,
of course, he'd agree that people should learn how to use it.)

Is syntax the same thing as goto? I'm not sure. If it is, then I am
like Knuth --- I feel sad when languages remove my ability to change
syntax. But surely I feel overwhelmed sometimes when there's too much
syntax created by the programmer who wrote the library that I'm trying
to understand.

(*) Footnotes

[1] Notes on structured concurrency
https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful

> (I suspect this will get garbaged in processing...)
>
> 4 5 ? 20 ? 52
>
> (it did... the first ? [in my display] is Greek lower case rho, the second
> question mark is... a question mark).

Like you, I see two question marks.

> 4 5 $RHO 20 ? 52
>
> Generate 20 random numbers in the range 1..52, no duplicates, reshape
> the vector into a 4x5 matrix.
>
> That just "dealt" four poker hands (needs logic to translate integer
> 1..52 into suit/rank).

That's wild. :-) Was this created by Brian Kernighan? It's hard to
believe. Oh, I think he wrote AMPL, wasn't it? A Mathematical
Programming Language, or something like that.
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
On Sun, 22 Aug 2021 16:28:12 -0300, Hope Rouselle <hrouselle@jevedi.com>
declaimed the following:


>That's wild. :-) Was this created by Brian Kernighan? It's hard to
>believe. Oh, I think he wrote AMPL, wasn't it? A Mathematical
>Programming Language, or something like that.

Kenneth Iverson, early 1960s for release, though he started in the late
50s (so a decade before Kernighan). I believe it started life as a notation
for publishing reports, not as an actual implemented language.

<https://en.wikipedia.org/wiki/APL_(programming_language)#Development_into_a_computer_programming_language>
{Hmmm, supposed to have influenced Matlab, S, and Wolfram/Mathematica}

One of the quirks is that one reads APL from right to left... cf:
<https://en.wikipedia.org/wiki/APL_(programming_language)#Pick_6_lottery_numbers>

You do not want to look down at the one-liner for Conway's Game of
Life.


--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed@ix.netcom.com http://wlfraed.microdiversity.freeddns.org/

--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
On 2021-08-22 16:28:12 -0300, Hope Rouselle wrote:
> I have a certain distaste for syntax too. For instance, I would much
> rather write and read ``first(ls)'' than ``ls[0]''.

Would you also prefer `twothousandthreehundredandtwentythird(ls)` over
`ls[2322]`?

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"
Re: on perhaps unloading modules? [ In reply to ]
"Peter J. Holzer" <hjp-python@hjp.at> writes:

> On 2021-08-22 16:28:12 -0300, Hope Rouselle wrote:
>> I have a certain distaste for syntax too. For instance, I would much
>> rather write and read ``first(ls)'' than ``ls[0]''.
>
> Would you also prefer `twothousandthreehundredandtwentythird(ls)` over
> `ls[2322]`?

Omg, I'm afraid you are serious! :-D We could use something like

list_get(ls, 2322).

Of course that ls[2322] is perfect fine.
--
https://mail.python.org/mailman/listinfo/python-list
Re: on perhaps unloading modules? [ In reply to ]
Dennis Lee Bieber <wlfraed@ix.netcom.com> writes:

> On Sun, 22 Aug 2021 16:28:12 -0300, Hope Rouselle <hrouselle@jevedi.com>
> declaimed the following:
>
>
>>That's wild. :-) Was this created by Brian Kernighan? It's hard to
>>believe. Oh, I think he wrote AMPL, wasn't it? A Mathematical
>>Programming Language, or something like that.
>
> Kenneth Iverson, early 1960s for release, though he started in the late
> 50s (so a decade before Kernighan). I believe it started life as a notation
> for publishing reports, not as an actual implemented language.
>
> <https://en.wikipedia.org/wiki/APL_(programming_language)#Development_into_a_computer_programming_language>
> {Hmmm, supposed to have influenced Matlab, S, and Wolfram/Mathematica}
>
> One of the quirks is that one reads APL from right to left... cf:
> <https://en.wikipedia.org/wiki/APL_(programming_language)#Pick_6_lottery_numbers>
>
> You do not want to look down at the one-liner for Conway's Game of
> Life.

I really do not. But I did. That's really wild. Dramatic. Speechless.
--
https://mail.python.org/mailman/listinfo/python-list