- Principle and history
- NZB files
- PAR files
- Newsbin Configuration
- Fixing Corruption with PARs
To download from usenet you must:
Principle and history
- Access to newsgroups
I use usenetserver.com, they have a affiliate scheme so if you want to sign up here's mine, I'd appreciate that! They have 101 days retention and you can have 10 connections, speed limit of 1MBit/s (which means 1mbit/s constant download speed)
- Newsgroup Client
I use newsbin pro and this intro will use that in examples
For corruption recovery
Newsgroups, or rather USEr NETwork newsgroups are not new, this is a common mistake by newbies under the impression that its a new alternative to p2p. Just because you've never heard of them, that is likely due to your lack of exposer to computing on a larger or historical scale.
In fact, usenet newgroups are actually far, far older than the WWW and the http protocol. Data is served over the nntp protocol, primarily designed for text and discussion at technical institutions and universities.. the main priniciple is large data servers that sync messages between other large data servers, with the messages split by groups.
The technology requires data posted by one user to propagate to your isps/hosts newsgroup server.
I won't repeat this all, you can find out detailed background at Wikipedia
. Just wanted to present the fact that usenet newsgroups are not new, and the posting of binaries was later developed on top of the network.
Usenet was originally created to distribute text content encoded in the 7-bit ASCII character set. With the help of programs that encode 8-bit values into ASCII, it became practical to distribute binary files content. Binary posts, due to their size and dubious copyright status, were in time restricted to specific newsgroups, making it easier for administrators to allow or disallow the traffic.
The oldest widely used encoding method is uuencode, from the Unix uucp package. In the late 1980s Usenet articles were often limited to 60,000 characters, and larger hard limits exist today. Files are therefore commonly split into sections that require reassembly by the reader.
With the header extensions and the Base64 and Quoted-Printable MIME encodings, there was a new generation of binary transport. In practice, MIME has seen increased adoption in text messages, but it is avoided for most binary attachments. Some operating systems with metadata attached to files use specialized encoding formats. For Mac OS, both Binhex and special MIME types are used.
Other lesser known encoding systems that may have been used at one time were BTOA, XX encoding, BOO, and USR encoding.
In an attempt to reduce file transfer times, an informal file encoding known as yEnc was introduced in 2001. It achieves about a 30% reduction in data transferred by assuming that most 8-bit characters can safely be transferred across the network without first encoding into the 7-bit ASCII space.
The standard method of uploading binary content to Usenet is to first archive the files into RAR archives (for large files usually in 20 MB or 50 MB parts) then create Parchive files. Parity files are used to recreate missing data. This is needed often, as not every part of the files reach a server. These are all then encoded into yEnc and uploaded to the selected binary groups.
Your concern here is likely binaries
Because of the limitation on message size, binary data is stored across several message posts and reconstructed by your news client.
Corruption is a distinct possibility because, for example, if your file is not on the servers anymore (retention ranges from days to several months, depending on how much you pay for access) you'll have messages that make up part of a file missing from the binary. There's a simple method of parity to recover corruption.NZB Files
NZBs were introduced to save bandwidth to get content. You do not need to subscribe to a group and download headers anymore to see what content is available. Like the introduction of ed2k links with edonkey2000 that meant you no longer had to rely on searching the network to find data, nzbs were introduced so you didn't have to download all the headers for messages and select and download to start a download from newsgroups.
An nzb file is similar to a torrent file in many ways, the file itself does not contain any useful information beyond when you maybe able to connect to get the data. A nzb may point to data that isn't there, or isn't on your isp server. Its up to you and your newsgroup client to acquire the data.
As far as nzbs go, you need to understand that the file just says which group the content is located, the message details, etc. Just metadata. You will need a client that can understand and download the content referred to in the nzb file.PAR files
The concept of parity should be okay with anyone in a technical or mathematical field. Its basic checksum maths.
Essentially when you have a release you can generate a certain amount of parity, a check on bit encoding, to allow for recovery from corruption. Here's a simple demonstration
Release MSG1: 0101010111011010011
Release MSG2: 1001010110100100011
Release MSG1: 0101010111011010011 (This is known)
Corrupt MSG2: ?001?1011010010?01? (There are several unknown bits [?])
Parity: 1100000001111110000 (This is known)
Recovered 2: 1001010110100100011
Easy, we don't know the first bit, but because the checksum of the par is 1 and msg1 is 0, it must be 1, etc.)
How par2 files work is the releaser states a certain amount of parity and upload those files for users to be able to repair any missing data they have. Checksums are done on parts of files called blocks. When you run the par2 file you'll checksum your files for corruption. If you have corruption, you'll be told you require xyz blocks. You'll need to get the number of corresponding par2 files. Blocks are indicated by the +xy at the end of parity files.
For example if you need 11 blocks, you can grab any number of par2 files that add up to 11blocks or more, ie.
+01 and +10, or just one +12, etc.Setting up newsbin pro
2) Go to options and set a download folder and a temp folder:
3) Setup a folder to load nzb files from, useful if your setting up nzb on a remote pc like me. You can put nzbs in here for them to load into newsbin. If your nzb is called horror.nzb, it will download the rars into a folder called horror. If your on a local pc you can just open nzb with newsbin for them to be added, but i'd still setup a folder to load nzb from.
4) Next add your server
5) In ye olden times you'd have to subscribe to groups, but with nzbs there's no need. You get nzbs from this site, http://www.nzbsrus.com
, etc. Anyway, lets add your server:
6) Setup your connections
7) NZBs downloads are added to the download tab
Scratch tab is where your par files from nzbs are added. To download from the scratch, right click a file and click Add to Download.NOTE:
If a file is incomplete, or refuses to complete, right click the file and click retry download. Try this a couple of times, if you have black blocks, right click and click "Assemble incompletes and delete". You'll be needing pars to fix the missing partsUsing pars to fix a download
1) Load the par2 file that should have downloaded with your release. You can either double click the file under the Files tab, or go to your completes folder and double click. You should have quickpar installed. For the first time you install, run through the options and check the monitor and auto-repair. When you double click you'll get a pop up that will say how many blocks you need after its scanned the files:
2) Download the required number of pars, here +5 were needed... so I could have got anything over that, or some files that added up to more. I just chose the +05:
3) The quickpar app will scan for the blocks in the folder, add them and begin to recover/repair:
4) When done all should be well:List of nzb siteshttp://forum.dead-donkey.com
(registration, daily limit of 5 downloads)http://www.binsearch.info
(old school like navigation of groups, click the boxes next to files and click create nzb)List of groupsHorrorMain Sourcesalt.binaries.multimedia.horror
* Popular and biggest horror newsgroupalt.binaries.dvd.midnightmovies
* DVDrsSmaller Groupsalt.binaries.sleazemoviesalt.binaries.ath
* Poorer Retention on ISPSalt.binaries.ratcaveScifialt.binaries.multimedia.scifialt.binaries.multimedia.scifi-and-fantasyHDTValt.binaries.hdtvalt.binaries.hdtv.repostalt.binaries.hdtv.x264Moviesalt.binaries.movies.xvidalt.binaries.moviesalt.binaries.movies.divxalt.binaries.movies.zeromoviesTValt.binaries.tv
If your looking to pay for signup, which is likely needed I use usenetserver.com, this is my affiliate link:http://www.usenetserver.com/?refer=154973
ISPs are rarely able to provide a decent fill rate, although you may be able to get things like tv episodes for the past day or so given their popularity.