Jump to content


Photo

Flashexpander for DM800HD OpenPli2.1

flash expander flashexpander

  • Please log in to reply
66 replies to this topic

Re: Flashexpander for DM800HD OpenPli2.1 #21 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 24 April 2013 - 13:24

Did you look at sub-page writes? That would be the first thing I'd look at to reduce overhead.
There's almost 2MB to gain on a 59MB ubi layer with 512 byte pages.

Re: Flashexpander for DM800HD OpenPli2.1 #22 Lost in Space

  • Senior Member
  • 876 posts

+69
Good

Posted 24 April 2013 - 13:46

Yes, because I could easily play with them with dFlash as there I can control what arguments to mkfs.ubifs are used for a backup.

Only problem ist that due to the squashfs images that DMM uses which are a handfull of big read only files the impact is not that great anymore.

 

Probably the results in PLi Images without the squashfs burden would be slightly better.



Re: Flashexpander for DM800HD OpenPli2.1 #23 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 24 April 2013 - 14:54

Yes, because I could easily play with them with dFlash as there I can control what arguments to mkfs.ubifs are used for a backup.

No, I actually mean subpage writes (in the nand driver), this is not something you can control with mkfs.ubifs arguments.

Re: Flashexpander for DM800HD OpenPli2.1 #24 Lost in Space

  • Senior Member
  • 876 posts

+69
Good

Posted 24 April 2013 - 15:05

Actually I found the reson why the broadcom nanddriver didn't work with ubifs (not spefifying writesize) so I spend a lot of time debugging it. 

 

And I also played with subpages as the dm8000 uses it but block2mtd doesn't fully supporting it (yet), but of simple ubifs extraction where I mainly use it that doesn't make any difference as it is a read only operation.

 

The ubinize for the small boxes specifies the -s 512 already and I tried some other values too as before DMM checked in the whole stuff I had to build kernels with the neded fixes and parameters on my own. It is not so easy to change these values as you then endup quickly with a kernel not suitable for standard jffs2 AND ubifs.

 

BTW the ubifs parameters of some of the non-dreamboxes use are also not for optimal performance and flash space consumption in my opinion, but due to their bigger size they seem not to care.

 

But for plain UBIFS discussion JTR Thread would be better as this is OT here.


Edited by gutemine, 24 April 2013 - 15:08.


Re: Flashexpander for DM800HD OpenPli2.1 #25 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 24 April 2013 - 15:10

Yes the dm8k uses it, but with 256MB the overhead is less of an issue.
The brcm nand controller does not have sub-page writing support for small-page nands, so it needs to be emulated in the driver.

Re: Flashexpander for DM800HD OpenPli2.1 #26 Lost in Space

  • Senior Member
  • 876 posts

+69
Good

Posted 24 April 2013 - 22:07

You can always lie on the geometry of the flash, read the original UBIFS Thread in the DMM board where I started with hardcoded sizes - remember I'm dumb :-)

 

With block2mtd it is the other way around - you can have writesizes down to 1 Byte, and there I added a simple patch to specify the writesize on the load, which allowed me to test the impact on size of different values without hacking too heavily the naddriver. I could have done the same for subpages, but it is not wort the effort.

 

Now it is the other way around, I respect the values that DMM uses in broadcom nand driver as they have to support it, and they respected the changerequests I did for block2mtd - as I'm the only one really using it anyway at the moment there was no risk involved for them.

 

block2mtd is much simplier and easier to use then nadsim, especially as OE 2.0 has this driver built into the kernel, which means you can now boot ubifs also from raw USB and Sata devices and unpack natively ubifs images with nfidump.

 

Show me any of your other great boxes which can do that at the moment :-)


Edited by gutemine, 24 April 2013 - 22:09.


Re: Flashexpander for DM800HD OpenPli2.1 #27 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 24 April 2013 - 23:33

I'm not talking about block2mtd or nandsim, just the flash driver.

Re: Flashexpander for DM800HD OpenPli2.1 #28 Rob van der Does

  • Senior Member
  • 7,766 posts

+184
Excellent

Posted 25 April 2013 - 05:16

Tens of MB for picons

 


EPG is stored in RAM. It does not use flash, or any other storage device at all.

But will be stored (in flash or any other device) as soon as E2 is stopped (albeit the size may be a bit smaller then the amount of RAM used).



Re: Flashexpander for DM800HD OpenPli2.1 #29 MiLo

  • PLi® Core member
  • 14,042 posts

+298
Excellent

Posted 25 April 2013 - 06:50



Tens of MB for picons

 

EPG is stored in RAM. It does not use flash, or any other storage device at all.


But will be stored (in flash or any other device) as soon as E2 is stopped (albeit the size may be a bit smaller then the amount of RAM used).


Which only makes it more suited for a good old harrdisk, even with the spinup time included, it will beat the internal flash when it comes to writing a big file.

The sie on disk is not a "bit" smaller. It's probably about half the size.
Real musicians never die - they just decompose

Re: Flashexpander for DM800HD OpenPli2.1 #30 gorski

  • Senior Member
  • 1,699 posts

+46
Good

Posted 25 April 2013 - 10:26

Me a lay person but I have a few questions...

 

Every time you use EPG HDD would have to start, if it goes from it, right? Similar with Picons etc. Isn't it better to have a fast USB stick with 2 partitions and have the FE on one and everything else on the other partition?

 

Also, a crucial, simple question: everybody acknowledges that 64MB is small for today's needs and a myriad of things we have on offer to test and play with. Why, then, is it difficult to acknowledge that FE has its place and legitimate function?

 

If one puts too much (esp. if it's the whole flash) on a USB stick - that can not be as fast as flash? So, why would anyone crack a nut with a 15kg hammer? It makes no sense!


Edited by gorski, 25 April 2013 - 10:29.

<span style='font-family: comic sans ms,cursive'>"Enlightenment is man's emergence from his self-incurred immaturity. Immaturity is the inability to use one's own understanding without the guidance of another. This immaturity is self-incurred if its cause is not lack of understanding, but lack of resolution and courage to use it without the guidance of another. The motto of enlightenment is therefore: Sapere aude! Have courage to use your own understanding!</span><br /> <br /><span style='font-family: comic sans ms,cursive'>Laziness and cowardice are the reasons why such a large proportion of men, even when nature has long emancipated them from alien guidance..." I. Kant, "Political writings" (1784)</span><br /> <br /><span style='font-family: comic sans ms,cursive'><a class='bbc_url' href='<a class='bbc_url' href='http://eserver.org/p...lightenment.txt'>http://eserver.org/p...ent.txt</a>'><a class='bbc_url' href='http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a>'>http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a></a> - the jolly text on Enlightenment, at the basis of Modernity...</span>

Re: Flashexpander for DM800HD OpenPli2.1 #31 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 25 April 2013 - 10:34

Every time you use EPG HDD would have to start, if it goes from it, right? Similar with Picons etc. Isn't it better to have a fast USB stick with 2 partitions and have the FE on one and everything else on the other partition?

No. For picons you are right. But epg is only written to hdd when e2 is stopped / restarted.
And only read when e2 starts again.
It's just a way to keep the data while e2 is not running. As long as e2 is running, all epg is kept only in ram.

Re: Flashexpander for DM800HD OpenPli2.1 #32 gorski

  • Senior Member
  • 1,699 posts

+46
Good

Posted 25 April 2013 - 10:50

OK, so if we need a USB stick for Picons, it seems reasonable to me to put the EPG there as well, together with SWAP and subtitles, 1Ch and so on... Practical, isn't it?

 

Also, if the stick is already there, why not use it for FE and get just a few folders over to the USB stick's first partition, so even if stuff is going somewhere to /usr folder it's not bad at all, if one comes close to filling up the flash (which one would, under "normal" circumstances, without using the FE)...

 

I mean, one of the first advices one gets, when one's start playing a bit more with one's box, is not to overdo it with CrossEPG, picons, skins, plugins, tools, scripts and so on because one will clog up the flash and use up all the memory, so...

 

Moreover, I am writing all this because I have tried it and verified that this is the case. Conversely, once I have done the FE properly (to a USB stick with 2 partitions, as I explained elsewhere), I have had a much more stable E2, working fine for long, even though I really stuffed it all, with maximum demand in all directions (lots of EPG downloaded for many providers, absolute loads of picons, a few skins, usually using the heavy Glass HD, with almost everything it has loaded/used, many tools, zillions of plugins and so forth).

 

I am no coder, not from IT etc. I do not fully understand what goes on in the background. But I know what I tested and the results I got.

 

The only thing I would suggest here is: test PB OpenPLi-based image of 1.11.2012, for instance, do it as I described, demand from it the most you can and see... Maybe there is something good to learn from it?


Edited by gorski, 25 April 2013 - 10:51.

<span style='font-family: comic sans ms,cursive'>"Enlightenment is man's emergence from his self-incurred immaturity. Immaturity is the inability to use one's own understanding without the guidance of another. This immaturity is self-incurred if its cause is not lack of understanding, but lack of resolution and courage to use it without the guidance of another. The motto of enlightenment is therefore: Sapere aude! Have courage to use your own understanding!</span><br /> <br /><span style='font-family: comic sans ms,cursive'>Laziness and cowardice are the reasons why such a large proportion of men, even when nature has long emancipated them from alien guidance..." I. Kant, "Political writings" (1784)</span><br /> <br /><span style='font-family: comic sans ms,cursive'><a class='bbc_url' href='<a class='bbc_url' href='http://eserver.org/p...lightenment.txt'>http://eserver.org/p...ent.txt</a>'><a class='bbc_url' href='http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a>'>http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a></a> - the jolly text on Enlightenment, at the basis of Modernity...</span>

Re: Flashexpander for DM800HD OpenPli2.1 #33 MiLo

  • PLi® Core member
  • 14,042 posts

+298
Excellent

Posted 25 April 2013 - 14:45

Also, a crucial, simple question: everybody acknowledges that 64MB is small for today's needs and a myriad of things we have on offer to test and play with. Why, then, is it difficult to acknowledge that FE has its place and legitimate function?

I'd be among the first to acknowledge that 64M is small.

And I'm all in favour of a flashexpander plugin. Now I can point anyone who runs out of internal flash to that plugin.
Real musicians never die - they just decompose

Re: Flashexpander for DM800HD OpenPli2.1 #34 MiLo

  • PLi® Core member
  • 14,042 posts

+298
Excellent

Posted 25 April 2013 - 14:54

If one puts too much (esp. if it's the whole flash) on a USB stick - that can not be as fast as flash?

So far, any attempt to move the root filesystem to USB resulted in something that booted only slower than the internal flash. This is probably limited by the USB controller.

The one exception was the dm7025. Because the CF reader was directly wired to the ATA controller, it was faster than its internal flash. The dm7025 boots in less than a minute when using a CF card, faster than the dm8000 with ubi (and a dual-core CPU at a 75% higher frequency).

When using another device to run everything from, you might as well move the whole root filesystem there, because the amount of extra space that will occupy is hardly worth the overhead of unionfs and similar solutions.
Real musicians never die - they just decompose

Re: Flashexpander for DM800HD OpenPli2.1 #35 Lost in Space

  • Senior Member
  • 876 posts

+69
Good

Posted 25 April 2013 - 14:59

FE is a nice plugin, but as it depends on udev/mdev,.. to be able to mount per  UUID instead of mounting /dev/sd* (which could change when devices are added/removed). It is also a little bit too much to always format teh device when you just need a directory there. This leads to all the confising patitioning advices. As there are still lots of things outside of /usr (like /lib) Flash space will still continuously decrease and defragment further. As it depends on the /etc/fstab entry for mounting /usr this entry could get lost and then your Flash image is inconsistent as parts of the flash image have been updated and the /usr not which is then mssing - this could lead to unbootable situation.

 

Finally it is hard to do full backups as the flash and the Extended /usr would need to be backuped and restored seperately. I therefore only allow to include /usr in a dFlash backup as nfi but if will not be flashable if the size is bigger then the flash (what is the goal of FE). So the backups can be only used form migration to other extension or Multiboot tools or have to be manually handled. Yes, this could be overcome or solved but this is not done yet and is a problem the users will face if something goes wrong - earlier or later.

 

FE is still a very usefull Plugin and mechatron did a great job to make it publically available so that people can develop it further and customize to their needs but as explained it has some serious design limitations which the users should be aware and understanding before enthusiastically using it.

 

And all my tools and even FE support SATA sticks and in wost case also harddisk for booting besides USB, because without detection time as the USB device need it boots still superiour to any UBIFS implementation but I agree that UBIFS is close.


Edited by gutemine, 25 April 2013 - 15:04.


Re: Flashexpander for DM800HD OpenPli2.1 #36 offnow

  • Senior Member
  • 56 posts

+8
Neutral

Posted 25 April 2013 - 15:50

What gorski is talking about has no need of udev or mdev and it is not expanding complete /usr only some folders of it to keep the speed of the image but provide more space.

 

Yes - it is booting slower - but this is not in my mind else my box is never off ;)

 

And frag of flash is not a real problem, because much kernel changes will end up in new flashing. And in this case complete backups are not a good idea - else if they were used elsewhere with things inside you want not to know...

 

At last - the FE is only to python translated stuff from dbox2 - have a talk to darkvolli, weazle and geko37 - maybe to ketschuss who tuned the hardware a long time ago...

You can read the handling in our doku ;)


hab keine Lust mehr...

Viel Spaß allen und lebt wohl


Re: Flashexpander for DM800HD OpenPli2.1 #37 gorski

  • Senior Member
  • 1,699 posts

+46
Good

Posted 25 April 2013 - 15:59

I used FE briefly and moved on to Expander, which is a part of PB Device Manager, and there things are different...

 

From DBox (where it started) to DMM to PB (Stibbich and co.) things have always changed a bit... Just to mention it...

 

Here is a bit from my "How to for PBNIGMA":

 

Originally, MMC device was installed in DBox2 by ketschuss and it seems geko37 and darkvolli wrote the plugin. Mechatron re-coded it for DB (/usr folder is moved to a USB device) and PB Team (Stibbich and co.) re-wrote the plugin and included it in all PB images. This PB plugin moves the /usr/share/enigma2 and /usr/lib/enigma2 folders to a chosen USB device/partition, so we never run out of space and our box will perform speedily and stably!


<span style='font-family: comic sans ms,cursive'>"Enlightenment is man's emergence from his self-incurred immaturity. Immaturity is the inability to use one's own understanding without the guidance of another. This immaturity is self-incurred if its cause is not lack of understanding, but lack of resolution and courage to use it without the guidance of another. The motto of enlightenment is therefore: Sapere aude! Have courage to use your own understanding!</span><br /> <br /><span style='font-family: comic sans ms,cursive'>Laziness and cowardice are the reasons why such a large proportion of men, even when nature has long emancipated them from alien guidance..." I. Kant, "Political writings" (1784)</span><br /> <br /><span style='font-family: comic sans ms,cursive'><a class='bbc_url' href='<a class='bbc_url' href='http://eserver.org/p...lightenment.txt'>http://eserver.org/p...ent.txt</a>'><a class='bbc_url' href='http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a>'>http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a></a> - the jolly text on Enlightenment, at the basis of Modernity...</span>

Re: Flashexpander for DM800HD OpenPli2.1 #38 gorski

  • Senior Member
  • 1,699 posts

+46
Good

Posted 25 April 2013 - 16:00

Hehe, Stibbich beat me to it by a second or three...... :D


<span style='font-family: comic sans ms,cursive'>"Enlightenment is man's emergence from his self-incurred immaturity. Immaturity is the inability to use one's own understanding without the guidance of another. This immaturity is self-incurred if its cause is not lack of understanding, but lack of resolution and courage to use it without the guidance of another. The motto of enlightenment is therefore: Sapere aude! Have courage to use your own understanding!</span><br /> <br /><span style='font-family: comic sans ms,cursive'>Laziness and cowardice are the reasons why such a large proportion of men, even when nature has long emancipated them from alien guidance..." I. Kant, "Political writings" (1784)</span><br /> <br /><span style='font-family: comic sans ms,cursive'><a class='bbc_url' href='<a class='bbc_url' href='http://eserver.org/p...lightenment.txt'>http://eserver.org/p...ent.txt</a>'><a class='bbc_url' href='http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a>'>http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a></a> - the jolly text on Enlightenment, at the basis of Modernity...</span>

Re: Flashexpander for DM800HD OpenPli2.1 #39 Lost in Space

  • Senior Member
  • 876 posts

+69
Good

Posted 25 April 2013 - 16:14

I think it is not about who wrote what and when. I didn't invent chroot booting either :-)

 

And it doesn't matter which and how many directories or files you kick out - it is about choosing the right ones with maximum impact and minimal harm.

 

But if you are not understanding that by doing a backup/restore=flashing you are defragmenting flash completely as all files are recompressed with mkfs.* and neatly aligned in the image then I'm sorry to confuse you.

 

It is different when you backup with nanddump, because then you dump the flash 1:1 as fragmented as it is.

 

Flodder for exmaple is a single binary doing the whole job and NOT changhing anything in the image and moving whole root to the flodder device. If you like it or not, it works too.


Edited by gutemine, 25 April 2013 - 16:18.


Re: Flashexpander for DM800HD OpenPli2.1 #40 gorski

  • Senior Member
  • 1,699 posts

+46
Good

Posted 25 April 2013 - 16:22

GM, you were just told what's what, so you do not put all in the same bag, as things have changed over time, from one coder to another. Also, you were told that in PB things do not work the way you "imagine" it. Respect it, for crying out loud... And there is no need for echo - what you just wrote (maximum impact etc.) Stibbich just explained to you - i.e. he described what he is doing with it...

 

Btw, I do not want to back up the whole thing. I do not want to run the whole thing off of a USB stick.

 

For me, after trying FE and Expander - PB works better, thanx....

 

Not to have to repeat it all, here is what I do exactly:

 

http://openpli.org/f...li/#entry344874


Edited by gorski, 25 April 2013 - 16:24.

<span style='font-family: comic sans ms,cursive'>"Enlightenment is man's emergence from his self-incurred immaturity. Immaturity is the inability to use one's own understanding without the guidance of another. This immaturity is self-incurred if its cause is not lack of understanding, but lack of resolution and courage to use it without the guidance of another. The motto of enlightenment is therefore: Sapere aude! Have courage to use your own understanding!</span><br /> <br /><span style='font-family: comic sans ms,cursive'>Laziness and cowardice are the reasons why such a large proportion of men, even when nature has long emancipated them from alien guidance..." I. Kant, "Political writings" (1784)</span><br /> <br /><span style='font-family: comic sans ms,cursive'><a class='bbc_url' href='<a class='bbc_url' href='http://eserver.org/p...lightenment.txt'>http://eserver.org/p...ent.txt</a>'><a class='bbc_url' href='http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a>'>http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a></a> - the jolly text on Enlightenment, at the basis of Modernity...</span>



Also tagged with one or more of these keywords: flash, expander, flashexpander

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users