Post new topic   Reply to topic
View previous topic Printable version Log in to check your private messages View next topic
Author Message
braveheartleoOffline
Post subject: Debian "sid" users beware of the dpkg 1.16.2 uploa  PostPosted: 14.03.2012, 07:09



Joined: 2011-04-08
Posts: 17

Status: Offline
      Quote:
Hi,

I'll be uploading dpkg 1.16.2 targeting unstable, by the end of
this weekend or beginning of next week the latest (after some final
polishing). Some pretty important points follow, the first section
in particular.


On-disk db damage
-----------------

The previous multiarch in-core db layout was bogus, resulting in a
possible inconsistent or broken on-disk db. If you are running any
dpkg derived from code that has never been in the main git repo (this
includes dpkg from the jenkins test builds [T], dpkg from experimental,
dpkg from Ubuntu, one of the personal pu/ branches, etc),
any of the
following might affect you.

[T] http://jenkins.grml.org/job/dpkg-test-source/

With those dpkg versions, when installing a M-A:same instance of
package P arch A, when a previous non-M-A:same version of package P
arch B was present in a config-files state, the installation would
incorrectly remove the control files on the infodb for the arch B
instance. You should check for any packages in the status db w/o
matching files under /var/lib/dpkg/info/.

Another effect of the bogus in-core db layout affected the disappearing
logic, so if you have been running any such dpkg versions, you should
also check in the logs that no package has been improperly disappeared,
although the installed files should still be present.

And finally (well, these are just some of the effects I've explicitly
looked for after revealing them through code review, there could be more
related to this, but I've not bothered after having reworked the in-core
db layout), upgrading from those versions to the new dpkg 1.16.2 might
cause the status db to get messsed up in some conditions. Before
upgrading, you should either downgrade to a non-multiarch enabled dpkg, or
make sure there's no package present (i.e. with status >= config-files)
with a mix of M-A:same and non-M-A:same instances. If there's such package
with two instances, the new dpkg will consider that a “cross-grade” and
as such replace one of them with the other, depending on the order they
are parsed, but leaving any control files untouched; if there's more than
two instances the new dpkg will outright refuse to parse such bogus and
inconsistent status db.

Do not file any bug report if you upgraded from a system with a damaged
db, w/o first reproducing it on a clean system.

(emphasis added by me for those possibly affected by this annoucement)

Read more here: http://lwn.net/Articles/486318/
 
 View user's profile Send private message  
Reply with quote Back to top
slamOffline
Post subject: RE: Debian "sid" users beware of the dpkg 1.16.2 u  PostPosted: 14.03.2012, 09:22
Team Member


Joined: 1970-01-01
Posts: 607
Location: w3
Status: Offline
This is NOT related to Aptosid/Debian Sid users, but to those who have not downloaded dpkg from the Debian main repositories only.
Greetings,
Chris

_________________
an operating system must operate
development is life
my Debian repo
 
 View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number 
Reply with quote Back to top
DonKultOffline
Post subject: Re: RE: Debian "sid" users beware of the dpkg 1.16  PostPosted: 14.03.2012, 10:03
Team Member


Joined: 2010-09-02
Posts: 480

Status: Offline
      slam wrote:
This is NOT related to Aptosid/Debian Sid users

Exactly, as long as you wasn't as crazy as 'apt-get install -t experimental dpkg' or even harder things like building it for yourself you can completely ignore the part about database-changes. That dpkg-internal stuff and nobody should really care about that (except the dpkg developers of course).

That said, if we really get this dpkg version soon in unstable there might be changes breaking other software. Especially avoid the temptation to activate MultiArch for now! Either recent apt-get version should be mostly fine - but mostly fine is not a good state for such a critical tool! If you have some self-made script interfacing with dpkg (like one to purge packages in conf-only state) be careful to check that they really still do what they do.

We (as in APT team) are currently waiting for dpkg to publish his implementation to adapt APT to whatever dpkg has changed last minute (the announcement includes some of the changes we had to deal with already, we don't know what comes up next as this work is again nowhere public accessible for testers…) and only then will ask the release team for a push-slot to unstable. I guess i will open a big party-post here then this transition is over, but this can easily require a few weeks…

On the pro-side: This gives the package maintainers in debian a chance to make their packages Multi-Arch ready, so that at the time you can activate it, you can also really us it. I might tell you then even why you might use it - or why you don't need it. Wink


If you feel like needing more info about how Multi-Arch made its way into debian, why i am a bit pissed and why the universe is still spinning, LWN has a nice (and mostly accurate) article about it: https://lwn.net/Articles/485263/ (sorry, today it's a paywall, but tomorrow the article will be free)

_________________
MfG. DonKult
"I never make stupid mistakes. Only very, very clever ones." ~ The Doctor
 
 View user's profile Send private message Visit poster's website  
Reply with quote Back to top
DeepDayzeOffline
Post subject: RE: Re: RE: Debian "sid" users beware of the dpkg  PostPosted: 14.03.2012, 17:46



Joined: 2010-09-11
Posts: 616
Location: USA
Status: Offline
So this soon will allow dpkg (and maybe later APT) to "crossgrade" the system from one arch to another (for example from i386 to amd64)? While I can see this still is a WIP, but the ability to crossgrade a system in place would be a great thing, especially if you move your install, say from an older i386 based arch to an x86_64 based one for example without having to reinstall.
 
 View user's profile Send private message  
Reply with quote Back to top
DonKultOffline
Post subject: RE: Re: RE: Debian "sid" users beware of the dpkg  PostPosted: 14.03.2012, 22:56
Team Member


Joined: 2010-09-02
Posts: 480

Status: Offline
Yeah, (simplification of) crossgrades is one of the plus-points. You can do this without multiarch already, but its a bit hard. Guillem lined out how to do it if you have dpkg with multiarch support installed (and an archive in which the needed packages are correctly multiarchified). The support to deal with this situation is implemented in APT, thats not the problem, what Guillem means is that you currently have to download the dependency of dpkg with APT and install them with dpkg directly. So what he means with APT missing cross-grade support is that you can't cross-grade an essential package like dpkg with APT (like: apt-get install dpkg:amd64) as APT will remove dpkg:i386 and then install dpkg:amd64 which obviously doesn't work as there is no dpkg anymore installed to install the new dpkg… (that dpkg can override itself [and other packages] is a new feature which we just do not support yet. As the theory behind is properly complicated and isn't defined yet [just implemented - maybe, i haven't seen public code yet] and we can live without it in 99% of all cases this will not be implemented soon: Crossgrades are easy to fuck up for a user, it doesn't really help if APT makes that easy if it just leads to more users fucking it up [as where is more to it than just installing a few packages - and that the only thing APT could do]).

Note to readers: If you have more questions marks over your head now after reading this than before reading it: Forget it. Multi-Arch implementation and the ideas and problems behind it are crazy shit. At the time debian wheezy will freeze (~3 months) we are hopefully at a point there i can say what an end-user and/or a developer can do with it without writing stuff like the above which isn't easy to understand. And - god is my witness - i tried hard to keep it easy, it could be a lot worser…


If you want to think about features of Multi-Arch, don't think about cross-grades. Aim lower for now: e.g. hassle-free (in terms of installation) skype, flash, $nonfree-app on amd64. wine for amd64 with out-of-the-box opengl correctly working. Developers can aim for simpler cross-building (even through that is stage2)…

_________________
MfG. DonKult
"I never make stupid mistakes. Only very, very clever ones." ~ The Doctor
 
 View user's profile Send private message Visit poster's website  
Reply with quote Back to top
DeepDayzeOffline
Post subject: Re: RE: Re: RE: Debian "sid" users beware of the d  PostPosted: 15.03.2012, 00:54



Joined: 2010-09-11
Posts: 616
Location: USA
Status: Offline
      DonKult wrote:
Yeah, (simplification of) crossgrades is one of the plus-points. You can do this without multiarch already, but its a bit hard. Guillem lined out how to do it if you have dpkg with multiarch support installed (and an archive in which the needed packages are correctly multiarchified). The support to deal with this situation is implemented in APT, thats not the problem, what Guillem means is that you currently have to download the dependency of dpkg with APT and install them with dpkg directly. So what he means with APT missing cross-grade support is that you can't cross-grade an essential package like dpkg with APT (like: apt-get install dpkg:amd64) as APT will remove dpkg:i386 and then install dpkg:amd64 which obviously doesn't work as there is no dpkg anymore installed to install the new dpkg… (that dpkg can override itself [and other packages] is a new feature which we just do not support yet. As the theory behind is properly complicated and isn't defined yet [just implemented - maybe, i haven't seen public code yet] and we can live without it in 99% of all cases this will not be implemented soon: Crossgrades are easy to fuck up for a user, it doesn't really help if APT makes that easy if it just leads to more users fucking it up [as where is more to it than just installing a few packages - and that the only thing APT could do]).

Note to readers: If you have more questions marks over your head now after reading this than before reading it: Forget it. Multi-Arch implementation and the ideas and problems behind it are crazy shit. At the time debian wheezy will freeze (~3 months) we are hopefully at a point there i can say what an end-user and/or a developer can do with it without writing stuff like the above which isn't easy to understand. And - god is my witness - i tried hard to keep it easy, it could be a lot worser…


If you want to think about features of Multi-Arch, don't think about cross-grades. Aim lower for now: e.g. hassle-free (in terms of installation) skype, flash, $nonfree-app on amd64. wine for amd64 with out-of-the-box opengl correctly working. Developers can aim for simpler cross-building (even through that is stage2)…


Will debian devs add multi-arch support for apt itself to handle the situation where the package with the old arch is removed first and then install the new arch with something more streamlined? That's the big missing piece right there
 
 View user's profile Send private message  
Reply with quote Back to top
DonKultOffline
Post subject: Re: RE: Re: RE: Debian "sid" users beware of the d  PostPosted: 15.03.2012, 10:40
Team Member


Joined: 2010-09-02
Posts: 480

Status: Offline
      DeepDayze wrote:
Will debian devs add multi-arch support for apt itself to handle the situation where the package with the old arch is removed first and then install the new arch with something more streamlined? That's the big missing piece right there

As said, we will at some point, but that is properly not tomorrow as it is just not that important now. This "problem" really only exists for a bit more than a handful of packages - most notable dpkg, other essentials and its dependencies - for all others there is simply no effective difference: If the package in question is e.g. iceweasel you will not really notice that APT removes iceweasel:i386 before installing iceweasel:amd64 (or you will only if you try to start iceweasel exactly between remove and install…). Supporting this "new" (= new for multiarch, for singlearch thats already the case) dpkg feature just means that APT can rely on the fact that dpkg will implicitly remove iceweael:i386 then it is asked to install iceweasel:amd64.
(if we imagine multi-arch to be a well-formed ladder, APT has implemented stage1 since 2010, got stage2 last year and cross-grades would be stage3. dpkg had not started climbing this ladder so far, now it did by a "rocketjump" to stage3 - but unfortunately the ladder melted down cause of the heat of the rocket, so we have stage1 with a few differences in shape, stage2 is considered partly-postpruned and stage3 is placed on top of this now slightly fragile ladder - dpkg now calls down to APT: "Hey bro, why aren't you at the top of the ladder anymore - too hot here? Wink" while we are busy to replace the parts of the ladder dpkg chose to melt down…)

Add that we have the problem that multiple dpkg versions are out there with different levels of multiarch support (=different implementations of these stages) which we can't really detect - so whatever we choose to assume to be true, we will piss of the rest of the world, so we chose for now to keep the requirement to a minimum so that we can satisfy both worlds - this also means we can't use this cross-grade enhancement, but that not really a big problem as you usually do cross-grades not on a daily basis…

Note: This "override" has no benefit for the user. I re-read what i wrote and it could be read like it would be save now to upgrade in 'init 5' or similar cause of that. That's not the case - the problem with that remains the same. This override just ensures that essentials are always available in multiarch as it is already the case in singlearch (aka now - to reuse the example above: You always have iceweasel installed which could be executed -- for essentials this means they will also work, iceweasel might not but thats a different story). You still have a decent chance to get a partly destroyed iceweasel in case you choose to let it run while you upgrade [e.g. iceweasel will try to detect that a new version is installed and display a restart message - but the images/text for this message is not necessarily included in the new version in the old place - if at all…]

_________________
MfG. DonKult
"I never make stupid mistakes. Only very, very clever ones." ~ The Doctor
 
 View user's profile Send private message Visit poster's website  
Reply with quote Back to top
Display posts from previous:     
Jump to:  
All times are GMT - 12 Hours
Post new topic   Reply to topic
View previous topic Printable version Log in to check your private messages View next topic
Powered by Zafenio