Web update library in python?

[Jorgen Bodde]

Hi all,

I want to provide my users the ability to download a repository from
the web, and after that check for updates. I thought of a mechanism
that could do that, but since there is patch and diff readily
available I wondered if there is a python solution that allows me to
download a file, and let the patch be applied locally. In the patch
there are binaries and sources I am not sure if patch can handle them
both though.

So here are the situations;

Downloading
1. User enters URL and local path where to store
2. Updated downloads the file
3. Either creates a new repository or patches an existing one

Check for updates
1. A web check is done if there is a newer version
2. File is downloaded, patch is applied

Plain zip file extraction is possible, but I will miss out on files
that might be deleted in future versions, but not yet on the client
side. Also local changes might need to be preserved although it is not
a show stopper if it doesn't.

I want to have a solution that is very easy for the user, so point and
click kind of work. Is there a tool / library around that can provide
me a base for this problem?

Thanks in advance!
- Jorgen

[Diez B. Roggisch]

Re: Web update library in python?

Jorgen Bodde wrote:

Hi all,
>
I want to provide my users the ability to download a repository from
the web, and after that check for updates. I thought of a mechanism
that could do that, but since there is patch and diff readily
available I wondered if there is a python solution that allows me to
download a file, and let the patch be applied locally. In the patch
there are binaries and sources I am not sure if patch can handle them
both though.
>
So here are the situations;
>
Downloading
1. User enters URL and local path where to store
2. Updated downloads the file
3. Either creates a new repository or patches an existing one
>
Check for updates
1. A web check is done if there is a newer version
2. File is downloaded, patch is applied
>
Plain zip file extraction is possible, but I will miss out on files
that might be deleted in future versions, but not yet on the client
side. Also local changes might need to be preserved although it is not
a show stopper if it doesn't.
>
I want to have a solution that is very easy for the user, so point and
click kind of work. Is there a tool / library around that can provide
me a base for this problem?

Subversion. With it's python-libs of course.

Diez

[A.T.Hofkamp]

Re: Web update library in python?

On 2007-11-20, Jorgen Bodde <jorgen.maillis t@gmail.comwrot e:

Hi all,
>
I want to provide my users the ability to download a repository from
the web, and after that check for updates. I thought of a mechanism
that could do that, but since there is patch and diff readily
available I wondered if there is a python solution that allows me to
download a file, and let the patch be applied locally. In the patch
there are binaries and sources I am not sure if patch can handle them
both though.

there is difflib

I want to have a solution that is very easy for the user, so point and
click kind of work. Is there a tool / library around that can provide
me a base for this problem?

However, from your description, it looks like you are trying to achieve what
many modern SCM tools do already out-of-the-box, so why are you trying to
re-invent the wheel?

For example, SVN handles anonymous download of working copies with HTTP,
distributed SCM tools (bzr, git, mercurial, to name a few) allow you to copy
complete repositories over the internet, and merge/update them (as well as
sending patches back by eg email).

Such a solution reduces your problem mostly to a one line script like 'svn
update'. You may want to add a nice GUI front-end around it (although given the
fact you are offering them a repository with sources (which, presumably is more
advanced than point-and-click), I don't really see the need for a
point-and-click solution).



Albert

[Jorgen Bodde]

Re: Web update library in python?

Hi All,

Thanks for the input. Concering 'egg' it's not a web update for python
per-ce but it should be written in python to easily interface with
(wx)Python. As I believe it 'egg' is only a dependency downloader and
installer for python scripts right?

As for subversion, this is not possible as I want to provide people
wihtout the ability to have a subversion server but only a plain web
site, to download a file that can upgrade a local work copy. Requiring
either SVN locally or server side is not really user friendly ..

Thanks for your input!
- Jorgen

On Nov 20, 2007 10:48 AM, Diez B. Roggisch <deets@nospam.w eb.dewrote:

Jorgen Bodde wrote:
>

Hi all,

I want to provide my users the ability to download a repository from
the web, and after that check for updates. I thought of a mechanism
that could do that, but since there is patch and diff readily
available I wondered if there is a python solution that allows me to
download a file, and let the patch be applied locally. In the patch
there are binaries and sources I am not sure if patch can handle them
both though.

So here are the situations;

Downloading
1. User enters URL and local path where to store
2. Updated downloads the file
3. Either creates a new repository or patches an existing one

Check for updates
1. A web check is done if there is a newer version
2. File is downloaded, patch is applied

Plain zip file extraction is possible, but I will miss out on files
that might be deleted in future versions, but not yet on the client
side. Also local changes might need to be preserved although it is not
a show stopper if it doesn't.

I want to have a solution that is very easy for the user, so point and
click kind of work. Is there a tool / library around that can provide
me a base for this problem?

>
Subversion. With it's python-libs of course.
>
Diez
--
>

http://mail.python.org/mailman/listinfo/python-list

>

[Jorgen Bodde]

Re: Web update library in python?

Hi A.T.Hofkamp,

Using svn still requires the people wanting to offer updates to have a
svn server installed, or use one of the svn services available and
that is simply too limiting.

I will explain a bit more where I will use this for. I am writing a
tool that takes a repository (e.g. a bunch of files with some XML to
tie it together) and generate something else based upon the user's
input configuration.

The 'repositories' can be created by anyone who likes to share their
code templates, and let the end user configure this template and
create a customized code base of it, on which they can code their
entire app. My tool supports incremental configuration which means if
there are updates to this repository, the user simply has to click
configure/generate again and merge the changes into their working
copy.

To provide others with a web update functionality from inside my tool,
all they should have to do is put a file online, containing a snapshot
(or delta snapshot files) of the repository that can be used. So all
they do is either click "UPDATE" or "DOWNLOAD" if the repository is
not yet present. For the maintainer of the repository it should be as
simple as making a snapshot of their directory and out comes a ZIP
file that can be put online for others to update from.

SVN is wonderful in resolving issues, but it requires everyone to
either use a service like berlios / sourceforge or be fortunate that
they have a SVN server set up somewhere. What I want, looks like a
plain down to earth update mechanism which is a bit more standardized
and it would be nice if there was already something like this called
PyWebUpdate, or PyUpdater.

For my tool it's key that the learning and usage curve is not very
steep. I can imagine on a web site you are simply instructed to enter
an url like "http://www.somesite.co m/repository/framework" inside my
tool, press DOWNLOAD / UPDATE and it will install it for you.

I guess where you can compare it with, is what Eclipse uses to update
or install their plugins ... This is not exactly re-inventing the
wheel as the majority of the users cannot use SVN (because they have
no place to store their repositories).
I really thought about SVN but I would not like end-users to be
bothered with maintaining or installing SVN or a SVN server, and have
intermediate .svn local files inside every directory ;-)

I hope this is a bit more clear,
With regards,
- Jorgen

[Diez B. Roggisch]

Re: Web update library in python?

Jorgen Bodde wrote:

Hi A.T.Hofkamp,
>
Using svn still requires the people wanting to offer updates to have a
svn server installed, or use one of the svn services available and
that is simply too limiting.
>
I will explain a bit more where I will use this for. I am writing a
tool that takes a repository (e.g. a bunch of files with some XML to
tie it together) and generate something else based upon the user's
input configuration.
>
The 'repositories' can be created by anyone who likes to share their
code templates, and let the end user configure this template and
create a customized code base of it, on which they can code their
entire app. My tool supports incremental configuration which means if
there are updates to this repository, the user simply has to click
configure/generate again and merge the changes into their working
copy.
>
To provide others with a web update functionality from inside my tool,
all they should have to do is put a file online, containing a snapshot
(or delta snapshot files) of the repository that can be used. So all
they do is either click "UPDATE" or "DOWNLOAD" if the repository is
not yet present. For the maintainer of the repository it should be as
simple as making a snapshot of their directory and out comes a ZIP
file that can be put online for others to update from.
>
SVN is wonderful in resolving issues, but it requires everyone to
either use a service like berlios / sourceforge or be fortunate that
they have a SVN server set up somewhere. What I want, looks like a
plain down to earth update mechanism which is a bit more standardized
and it would be nice if there was already something like this called
PyWebUpdate, or PyUpdater.
>
For my tool it's key that the learning and usage curve is not very
steep. I can imagine on a web site you are simply instructed to enter
an url like "http://www.somesite.co m/repository/framework" inside my
tool, press DOWNLOAD / UPDATE and it will install it for you.
>
I guess where you can compare it with, is what Eclipse uses to update
or install their plugins ... This is not exactly re-inventing the
wheel as the majority of the users cannot use SVN (because they have
no place to store their repositories).
I really thought about SVN but I would not like end-users to be
bothered with maintaining or installing SVN or a SVN server, and have
intermediate .svn local files inside every directory ;-)
>
I hope this is a bit more clear,

You won't come far with this. Sorry to say so, but the limitations you put
on yourself (whilst understandably so) are to much.

The reason is simply that without any server-side mechanism that at _least_
allows for file-locking (something plain HTTP doesn't, nor does FTP), you
can't possibly make this work, as different concurrent requests of users
will end up corrupting the data.

SVN works through Apache via HTTP. I REALLY believe you will spare yourself
_mucho_ troubles utilizing that.

Diez

[Jorgen Bodde]

Re: Web update library in python?

Hi Diez ,

I fail to see that. If I am the one that can only put new "archive
files" on my server by FTP or HTTP upload or whatever, and I update
the file which contains the information about what updates are present
on my site as last, there is no data corruption or loss of data.

There are two roles:

Repository maintainer:
-----
- Developer of the 'repository' creates a snapshot
- This archive is uploaded on his private site
- A file that accompanies the archive containing the list of updates
is sent last

Repository updater:
-----
- Downloads the accompanying file
- If the accompanying file is valid (I agree upon uploading this small
file can be downloaded, but since it is a small text file I would not
consider this a huge risk)
- Downloads the archive that was put there earlier

It might not be a 100% secure methodology, but I won't claim I get
thousands of hits per hour to begin with, I am lucky if someone is
willing to try my tool to begin with ;-)

If all fails I can temporarily rename the directory on the server
side, upload the files, and rename it back so that no corrupted files
are downloaded. But even it the small 'dictionary' file is corrupted,
the install process can abort and the user is asked to try again
later.

But I guess I am defending my way of solving the issue while the main
question was if there was anything remotely similar to what I would
need, besides using SVN that is ;-)

Thank you for your input!
- Jorgen

[Jorgen Bodde]

Re: Web update library in python?

I must add, the way I was originally attempting to solve this is
issuing delta files per upgrade cycle so that only changes are placed
in a new ZIP file. The delta zip file will always be a new file on the
site / ftp location. I do understand that if you are re-using the same
archive file all the time, people might download the file at the same
time I am uploading a new version.

So the bottleneck indeed is with the small accompanying file the
updater should download to see if there are changes needed to
download. If that file is updated while it's downloaded, it should
somehow be detected.

thanks for pointing that out..

Regards,
- Jorgen

On Nov 20, 2007 12:41 PM, Jorgen Bodde <jorgen.maillis t@gmail.comwrot e:

Hi Diez ,
>
I fail to see that. If I am the one that can only put new "archive
files" on my server by FTP or HTTP upload or whatever, and I update
the file which contains the information about what updates are present
on my site as last, there is no data corruption or loss of data.
>
There are two roles:
>
Repository maintainer:
-----
- Developer of the 'repository' creates a snapshot
- This archive is uploaded on his private site
- A file that accompanies the archive containing the list of updates
is sent last
>
Repository updater:
-----
- Downloads the accompanying file
- If the accompanying file is valid (I agree upon uploading this small
file can be downloaded, but since it is a small text file I would not
consider this a huge risk)
- Downloads the archive that was put there earlier
>
It might not be a 100% secure methodology, but I won't claim I get
thousands of hits per hour to begin with, I am lucky if someone is
willing to try my tool to begin with ;-)
>
If all fails I can temporarily rename the directory on the server
side, upload the files, and rename it back so that no corrupted files
are downloaded. But even it the small 'dictionary' file is corrupted,
the install process can abort and the user is asked to try again
later.
>
But I guess I am defending my way of solving the issue while the main
question was if there was anything remotely similar to what I would
need, besides using SVN that is ;-)
>
Thank you for your input!
- Jorgen
>

[Paul Boddie]

Re: Web update library in python?

On 20 Nov, 12:17, "Diez B. Roggisch" <de...@nospam.w eb.dewrote:

>
The reason is simply that without any server-side mechanism that at _least_
allows for file-locking (something plain HTTP doesn't, nor does FTP), you
can't possibly make this work, as different concurrent requests of users
will end up corrupting the data.
>
SVN works through Apache via HTTP. I REALLY believe you will spare yourself
_mucho_ troubles utilizing that.

Or perhaps something (else) which provides a comprehensive WebDAV
implementation? I believe Subversion exports repositories via HTTP
using various DAV-related methods, although I haven't needed to
explicitly interact with Subversion repositories in such a way.
Perhaps just exposing part of a filesystem using Apache plus mod_dav
might be a sufficient alternative.

Of course, as has already been said, lots of version control systems
provide Web-based interfaces, and there are various graphical
repository browsers which should know how to interact with these
interfaces. Moreover, some distributed version control systems are
able to understand repositories which are published as static files on
the Web, being able to navigate to the appropriate resources without
any dynamic magic happening on the server; Mercurial has support for
this, although I imagine that it's really a read-only thing:

http://www.selenic.com/mercurial/wiki/index.cgi/StaticHTTP

Ultimately, if you're pushing things back to the server and it's on a
"per change" basis, then a plain Web server isn't going to be
sufficient. Otherwise, if users push their changes out only as new
branches (in Mercurial terminology), they could upload such a branch
in a single transaction (copy an archive to the server via FTP/scp,
unpack it, move it into place) and have it exposed to the other users
via a mechanism like the one mentioned above.

Paul

[A.T.Hofkamp]

Re: Web update library in python?

On 2007-11-20, Jorgen Bodde <jorgen.maillis t@gmail.comwrot e:

The 'repositories' can be created by anyone who likes to share their
code templates, and let the end user configure this template and
create a customized code base of it, on which they can code their
entire app. My tool supports incremental configuration which means if
there are updates to this repository, the user simply has to click
configure/generate again and merge the changes into their working
copy.

SVN seems not useful in your situation, exactly because of the assumption of a
single authorative repository that everybody uses, as you described.



Instead, I think you should have a look at distributed SCM tools.

Distributed SCMs do not have the concept of a central repo (that's why they are
called 'distributed'), everybody can start their own repo (or copy one from
elsewhere), hack away, and offer their changes to others.
The system manages exchange of patches between the repositories.

I have currently very limited experience with distributed SCMs. I just started
experimenting locally with bzr.

Please note that 'bzr repository' is comparable to a 'svn working copy'. There
are no doubt subtle and not-so-subtle differences, but as I said, I have
limited experience...


The problem that I solved was that I wanted to have several personal
sub-projects inside a SVN-based project (that I do not own and have no write
access to). The changes of my sub-projects are then feed back into the main
project and (hopefully) committed.
Of course, these diffs should be against a recent SVN trunk of the project.

I created a project_trunk directory with the anonymous SVN checkout. On top
of svn, I added bzr management.
(after 'svn update' I add/commit the changes also in bzr in the same directory).
This directory now functions as an authorative source of bzr updates.

Now, each personal sub-project is a copy of that bzr repo. I can do arbitrary
code hacking, committing, running diff etc in each sub-project.
I can also merge changes from the project_trunk directory with 'bzr merge' to
pull in new updates from the SVN project.

Although I haven't tried it, it should also be possible to copy/merge changes
between my personal sub-projects.


[hmm, while answering your post, I read some of the bzr website, and found that
bzr has a standard tool for doing what I do... nice ]

I really thought about SVN but I would not like end-users to be
bothered with maintaining or installing SVN or a SVN server, and have
intermediate .svn local files inside every directory ;-)

Now in your case, my 'project_trunk' would be your central web-site, and my
personal sub-projects would be your users.
Of course, the bandwidth between your website and users is a bit smaller than
my local file system bandwidth, but given the fact that these SCMs are designed
to work in a distributed way across the Internet, so I would expect them to
handle this.



Albert

[Bjoern Schliessmann]

Re: Web update library in python?

Jorgen Bodde wrote:

There are two roles:
>
Repository maintainer:
-----
- Developer of the 'repository' creates a snapshot
- This archive is uploaded on his private site
- A file that accompanies the archive containing the list of
updates is sent last
>
Repository updater:
-----
- Downloads the accompanying file
- If the accompanying file is valid (I agree upon uploading this
small file can be downloaded, but since it is a small text file I
would not consider this a huge risk)
- Downloads the archive that was put there earlier

Still, subversion would let you accomplish this. Also I wonder where
you'd like to save state information in downloaded copies -- that's
what subversion does in .svn directories. You have to save the
state to update economically.

It might not be a 100% secure methodology, but I won't claim I get
thousands of hits per hour to begin with,

Do you mean "security" or "reliabilit y"?

But I guess I am defending my way of solving the issue while the
main question was if there was anything remotely similar to what I
would need, besides using SVN that is ;-)

I don't really understand this.

Regards,


Björn

--
BOFH excuse #150:

Arcserve crashed the server again.

[Bjoern Schliessmann]

Re: Web update library in python?

Jorgen Bodde wrote:

As for subversion, this is not possible as I want to provide
people wihtout the ability to have a subversion server but only a
plain web site,

What's a "plain web site"? Could mod_python be on it?

to download a file that can upgrade a local work copy.

As others mentioned, there is quite a risk that your files get out
of sync if someone downloads while you upload. The local copy will
be partly old, partly new. If the server can't do locking, you'll
need to control everything client-side. Which means hashing all
files and comparing against your control file. With much diff
downloading this could get really messy, I think.

Requiring either SVN locally or server side is not really user
friendly ..

Requiring _your_ tool locally is? :) If I were your client, I'd
prefer subversion (since I use it frequently).

Regards,


Björn

--
BOFH excuse #25:

Decreasing electron flux