›
This forum is in permanent archive mode. Our new active community can be found here.

Scripts to download the entire run of geeknights

malzraamalzraa
in GeekNights
In case anyone else wants to grab the entire (at least as far as I can tell) run of the show, since the RSS feed doesn't go all the way back, I've written two short and somewhat kludgey bash scripts for downloading all of the episode mp3 files. The gist containing them is here:

https://gist.github.com/bwkeller/075ec6179daa29ecf4d5

I'm curious to see if anyone knows of episodes I've missed. These fetch everything past 31-10-2005.

Comments

  • SuperPichuSuperPichu
    How large is this?
  • StarfoxStarfox
    edited January 2015
    I think this would work too, but Scott might get grumpy about it.
    wget -rH -Dtraffic.libsyn.com,apreche.net -A mp3 http://www.frontrowcrew.com

    wget gets things from the web
    -r makes it go recursively
    -H makes it recurse over hosts
    -D limits it to recursing onto traffic.libsyn.com and apreche.net
    -A makes it only get mp3's
    http://www.frontrowcrew.com makes it start on www.frontrowcrew.com

    It would crawl the whole site, everything reachable by link anyway. I don't know what kind of server it's running on, but you might DOS it.
    Post edited by Starfox on
  • DaikunDaikun
    Why would anyone do this?
  • StarfoxStarfox
    Starfox said:

    wget -rH -Dtraffic.libsyn.com,apreche.net -A mp3 http://www.frontrowcrew.com

    Of course, this won't get the beta episodes.

    Unless they're linked to somewhere, that is...
  • SuperPichuSuperPichu
    Starfox said:

    Of course, this won't get the beta episodes.

    Unless they're linked to somewhere, that is...

    I don't think anyone on the forums found those without being immediately sworn to secrecy by scrym
  • StarfoxStarfox
    SuperPichu said:

    I don't think anyone on the forums found those without being immediately sworn to secrecy by scrym

    I don't see either of them being the censorship types. I mean, if they are hosted at a publicly-accessible URL, they don't have much standing to complain when I immediately put them on a torrent, youtube, and soundcloud.
  • Hitman HartHitman Hart
    edited January 2015
    No, they just think that we all think exactly like how they think and know their entire web history, so it's EASY to find them. ;)
    Post edited by Hitman Hart on
  • malzraamalzraa
    SuperPichu said:

    How large is this?

    41GB!

  • malzraamalzraa
    edited January 2015
    Starfox said:

    I think this would work too, but Scott might get grumpy about it.
    wget -rH -Dtraffic.libsyn.com,apreche.net -A mp3 http://www.frontrowcrew.com

    wget gets things from the web
    -r makes it go recursively
    -H makes it recurse over hosts
    -D limits it to recursing onto traffic.libsyn.com and apreche.net
    -A makes it only get mp3's
    http://www.frontrowcrew.com makes it start on www.frontrowcrew.com

    It would crawl the whole site, everything reachable by link anyway. I don't know what kind of server it's running on, but you might DOS it.

    Beautiful, a one-liner!
    Post edited by malzraa on
  • AmpAmp
    So for the lay man would this turn up the fabled lost episodes?
  • ninjarabbininjarabbi
    No, I don't think they are hosted with all of the normal episodes.

    I'm not sure how libsyn's billing is, but downloading 41 gigs at once might be kind of a dick move as far as bandwidth fees and the aformentioned DOS. If you do, however, and you intend to keep them all I suggest making a bittorrent sync of them for the few other people who might be interested, so we don't have several people hitting their server really hard.
  • StarfoxStarfox
    Amp said:

    So for the lay man would this turn up the fabled lost episodes?

    No, they're somewhere else.

    I thought R&S used libsyn because they charge a flat rate no matter how much bandwidth you use. And I'm sure they have plenty of tubes to handle ~40 GB. But then you're downloading 40 gigs of two guys going on about NYC and bikes.
  • RymRym
    Download all you want. We have infinite bandwidth. The aggregate one-off downloads of the 1000ish episode backlog is already more bandwidth than you'd think!
  • Luke BurrageLuke Burrage
    edited February 2015
    Similar to my podcast too. I released only one podcast last month (as there's no set schedule), but from Jan 1st to Feb 1st, more than half of the bandwidth is spent on files outside the top 30 most popular.
    #reqs %bytes last time file

    3180 7.50% Feb/ 1/15 12:43 AM /audio/SFBRP #261 - Neal Asher - The Voyage of the Sable Keech.mp3
    2603 9.78% Feb/ 1/15 12:30 AM /audio/SFBRP #260 - Iain M Banks - The Hydrogen Sonata.mp3
    1647 3.80% Feb/ 1/15 12:28 AM /audio/SFBRP #259 - Brandon Sanderson - Words of Radiance.mp3
    1229 2.42% Jan/31/15 11:08 PM /audio/SFBRP #257 - Interstellar.mp3
    674 Feb/ 1/15 12:45 AM /
    650 1.36% Jan/31/15 11:13 PM /audio/SFBRP #256 - Peter F Hamilton - Pandora's Star and Judas Unchained.mp3
    566 0.19% Jan/31/15 11:46 PM /audio/SFBRP #193 - Lois McMaster Bujold - Barrayar.mp3
    551 0.23% Jan/31/15 11:50 PM /audio/SFBRP #214 - Alastair Reynolds - On the Steel Breeze.mp3
    485 Feb/ 1/15 12:09 AM /robots.txt
    475 0.84% Jan/31/15 11:09 PM /audio/SFBRP #252 - The Hugulas - Every novel that won both a Hugo and Nebula award.mp3
    457 0.69% Jan/31/15 11:14 PM /audio/SFBRP #258 - Nancy Kress - Beggars in Spain.mp3
    449 0.99% Jan/31/15 7:46 AM /audio/SFBRP #190 - Steven Erikson - Malazan Book of the Fallen #1 - Gardens of the Moon.mp3
    441 0.47% Jan/31/15 10:59 PM /audio/SFBRP #238 - David Brin - Startide Rising.mp3
    434 2.03% Jan/31/15 11:11 PM /audio/SFBRP #255 - Ernest Cline - Ready Player One.mp3
    388 0.18% Jan/31/15 7:45 AM /audio/SFBRP #057 - John Scalzi - Old Man's War.mp3
    364 0.28% Feb/ 1/15 12:58 AM /audio/SFBRP #227 - Rudy Rucker - Software and Wetware.mp3
    361 1.44% Jan/31/15 7:04 PM /audio/SFBRP #180 - Philip K Dick - The Man in the High Castle.mp3
    344 0.30% Jan/31/15 10:14 PM /audio/SFBRP #154 - Ted Chiang - Stories of Your Life and Others.mp3
    342 0.06% Jan/31/15 7:19 AM /audio/SFBRP #062 - Stephen Donaldson - The Real Story.mp3
    329 1.56% Jan/31/15 7:42 AM /audio/SFBRP #174 - Frank Herbert - Dune.mp3
    319 1.55% Jan/31/15 1:05 PM /audio/SFBRP #200 - Top Science Fiction Novels and Series.mp3
    317 0.81% Jan/31/15 11:06 PM /audio/SFBRP #254 - Peter F Hamilton - The Abyss Beyond Dreams.mp3
    287 1.00% Jan/31/15 11:07 PM /audio/SFBRP #237 - Larry Niven - Ringworld.mp3
    287 0.73% Jan/31/15 11:05 PM /audio/SFBRP #247 - Isaac Asimov - The Gods Themselves.mp3
    278 0.89% Jan/31/15 11:14 PM /audio/SFBRP #250 - Listener Suggestions for the Special 250th Episode.mp3
    278 0.54% Jan/31/15 11:01 PM /audio/SFBRP #253 - Jeff VanderMeer - Southern Reach #1 - Annihilation.mp3
    267 1.39% Jan/31/15 1:14 PM /audio/SFBRP #212 - Andy Weir - The Martian.mp3
    254 0.65% Jan/31/15 7:33 AM /audio/SFBRP #099 - Patrick Rothfuss - The Name of the Wind.mp3
    251 0.51% Jan/31/15 7:48 AM /audio/SFBRP #114 - Brandon Sanderson - Mistborn #1 - The Final Empire.mp3
    251 0.40% Jan/31/15 7:44 AM /audio/SFBRP #120 - Joe Abercrombie - The Blade Itself.mp3

    20802 57.43% Feb/ 1/15 12:55 AM [not listed: 268 files]
    Post edited by Luke Burrage on
  • SuperPichuSuperPichu
    If anyone has the urge to listen from the beginning (as I currently do due to the amount of time I spend sitting bored at a desk now) I generated RSS feeds for the entire (non-beta) run of GeekNights. I had to split it in half due to size.
    Full A
    Full B
  • ColdguyColdguy
    SuperPichu said:

    If anyone has the urge to listen from the beginning (as I currently do due to the amount of time I spend sitting bored at a desk now) I generated RSS feeds for the entire (non-beta) run of GeekNights. I had to split it in half due to size.
    Full A
    Full B

    Are you going to make show notes so that future generations can get a summary.
  • SuperPichuSuperPichu
    Coldguy said:

    SuperPichu said:

    If anyone has the urge to listen from the beginning (as I currently do due to the amount of time I spend sitting bored at a desk now) I generated RSS feeds for the entire (non-beta) run of GeekNights. I had to split it in half due to size.
    Full A
    Full B

    Are you going to make show notes so that future generations can get a summary.
    Ha ha. Nah it currently just loads scrapes the actual show notes.
  • Lt. ChibiLt. Chibi
    SuperPichu said:

    If anyone has the urge to listen from the beginning (as I currently do due to the amount of time I spend sitting bored at a desk now) I generated RSS feeds for the entire (non-beta) run of GeekNights. I had to split it in half due to size.
    Full A
    Full B

    Thank you, this will aid in my desk-sitting adventures as well.

    Would it be rude or incorrect to set up a torrent of the episodes? Maybe by year? I don't know how much Rym or Scott care about keeping track of downloads, but it would be easier than downloading them one by one.

  • ColdguyColdguy
    Lt. Chibi said:

    SuperPichu said:

    If anyone has the urge to listen from the beginning (as I currently do due to the amount of time I spend sitting bored at a desk now) I generated RSS feeds for the entire (non-beta) run of GeekNights. I had to split it in half due to size.
    Full A
    Full B

    Thank you, this will aid in my desk-sitting adventures as well.

    Would it be rude or incorrect to set up a torrent of the episodes? Maybe by year? I don't know how much Rym or Scott care about keeping track of downloads, but it would be easier than downloading them one by one.

    It's creative commons license as lo g ad you give credit to them they should be fine.
  • ninjarabbininjarabbi
    Someone should put them all on archive.org as well.
  • Josh BytesJosh Bytes
    edited October 2015
    Whoops wrong thread.
    Post edited by Josh Bytes on
Sign In or Register to comment.