Python Resources on the Internet

CR4 Home | General Discussion | General Discussion

16 comments

Rate

Bayes The Engineer Join Date: Feb 2005 Location: Albany, New York Posts: 5060 Good Answers: 129	Python Resources on the Internet 07/13/2016 7:26 AM
	Hi all, I'm trying to find some sample Python code for downloading the content of webpages. I'm new to Python, though have some skills (I understand dictionaries, lists, tuples, functions, not great with classes). I'm looking for a place on the internet where I can go to search for python code which I can then fiddle with an figure out. My searches so far usually take me to python.org (helpful) or stackoverflow (rarely helpful). If anyone has any go to spots they find helpful, please let me know. Thanks!
	Register to Reply

Interested in this topic? By joining CR4 you can "subscribe" to
this discussion and receive notification when new comments are added.

Join CR4, The Engineer's Place for News and Discussion!

"Almost" Good Answers:

Check out these comments that don't yet have enough votes to be "official" good answers and, if you agree with them, vote them!

#10 "Re: Python Resources on the Internet" by SolarEagle on 07/13/2016 12:36 PM (score 1)

JohnDG Guru Join Date: May 2006 Location: Reading, Berkshire, UK. Going under cover. Posts: 9684 Good Answers: 468	#1 Re: Python Resources on the Internet 07/13/2016 7:52 AM
	https://www.google.co.uk/search?q=sample+Python+code+for+downloading+the+content+of+webpages&ie=utf-8&oe=utf-8&client=firefox-b&gfe_rd=cr&ei=ASuGV463E83W8gf0joDwAg __________________ "Love justice, you who rule the world" - Dante Alighieri
	Register to Reply Score 1 for Off Topic

Bayes The Engineer Join Date: Feb 2005 Location: Albany, New York Posts: 5060 Good Answers: 129	#2 In reply to #1 Re: Python Resources on the Internet 07/13/2016 8:00 AM
	http://www.justfuckinggoogleit.com/
	Register to Reply Off Topic (Score 5)

JohnDG Guru Join Date: May 2006 Location: Reading, Berkshire, UK. Going under cover. Posts: 9684 Good Answers: 468	#3 In reply to #2 Re: Python Resources on the Internet 07/13/2016 8:17 AM
	I justfuckingdid. Did you? __________________ "Love justice, you who rule the world" - Dante Alighieri
	Register to Reply Off Topic (Score 5)

Bayes The Engineer Join Date: Feb 2005 Location: Albany, New York Posts: 5060 Good Answers: 129	#4 In reply to #3 Re: Python Resources on the Internet 07/13/2016 8:31 AM
	Of course. In my OP I asked if anyone knew of any resources. I explicitly stated I had done searches and what the results were. You then sent me a link to a search result with those results. How is that helpful? I don't think it is unreasonable for me to come to a forum and ask for help. I'm clearly not in school trying to get homework answers, nor am I a programmer so this isn't work related. In truth, I just wanted to figure out how to analyze CR4 and provide interesting posts since people have responded well to those in the past. I have an idea where I would like to get the text, remove stop words, and find out what words come up most frequently on CR4 and then post the results for all to see. But it seems to me like you took it upon yourself to give me a sarcastic response which quite frankly I didn't appreciate. That said, I probably should have just let it go. My response wasn't very mature.
	Register to Reply Off Topic (Score 5)

JohnDG Guru Join Date: May 2006 Location: Reading, Berkshire, UK. Going under cover. Posts: 9684 Good Answers: 468	#5 In reply to #4 Re: Python Resources on the Internet 07/13/2016 8:57 AM
	If you just want to get at the CR4 source, use "Tools->Developer->Page Source" (in Firefox). That's what I did when I first joined (except it wasn't Firefox), and made one or two suggestions to the author - who hasn't posted on here since 2011 - to help navigation (which were adopted). (BTW the search does yield over 2,000,000 hits - surely you could find something in there?). __________________ "Love justice, you who rule the world" - Dante Alighieri
	Register to Reply Off Topic (Score 5)

Bayes The Engineer Join Date: Feb 2005 Location: Albany, New York Posts: 5060 Good Answers: 129	#6 In reply to #5 Re: Python Resources on the Internet 07/13/2016 9:07 AM
	So in your mind, it is better for me to go through 2,000,000 google results rather than ask if someone more experienced has a resource that they like to use that might help me. I'm sorry, I don't agree with that logic. And your suggestion doesn't help me with Python. I know how to view the html source. Basically I asked "Does anyone know of any good Python resources on the web" and you've responded by showing me how to do a google search and explaining how to view the page html from a browser.
	Register to Reply Off Topic (Score 5)

JohnDG Guru Join Date: May 2006 Location: Reading, Berkshire, UK. Going under cover. Posts: 9684 Good Answers: 468	#7 In reply to #6 Re: Python Resources on the Internet 07/13/2016 9:09 AM
	<unsubscribe> __________________ "Love justice, you who rule the world" - Dante Alighieri
	Register to Reply Off Topic (Score 6)

Doggerel Associate Join Date: Sep 2010 Location: Old Midwest U.S.A. Posts: 44 Good Answers: 3	#8 Re: Python Resources on the Internet 07/13/2016 11:28 AM
	I am not sure if this is what you are searching for but Paul Lutus' website includes several Python programs he has developed. I have no affiliation with Mr. Lutus and this is not an endorsement.
	Register to Reply

Bayes The Engineer Join Date: Feb 2005 Location: Albany, New York Posts: 5060 Good Answers: 129	#9 In reply to #8 Re: Python Resources on the Internet 07/13/2016 12:19 PM
	It didn't have what I was looking for but it is exactly the sort of website I'm looking for. I'm looking for working code (though it can be general and basic) that can take the text off a webpage and save it to a txt file with a brief explanation of what it's doing. I can then build it up to what I want once I understand how it works. The problem with me is, unlike some people who can just read programming syntax explanations and understand it, I need a basic bit of code that works and I mess around with it till I understand it. Then I go back and read the tutorials and I can understand what it's saying.
	Register to Reply

Bayes The Engineer Join Date: Feb 2005 Location: Albany, New York Posts: 5060 Good Answers: 129	#11 In reply to #8 Re: Python Resources on the Internet 07/14/2016 8:20 AM
	I appreciate the link, unfortunately that's a bit to specific and complex. I'm looking more for a basic program that I can build up. Thanks though.
	Register to Reply

SolarEagle Guru Join Date: Mar 2007 Location: by the beach in Florida Posts: 33392 Good Answers: 1817	#10 Re: Python Resources on the Internet 07/13/2016 12:36 PM
	https://wiki.python.org/moin/SimplePrograms http://www.java2s.com/Code/Python/CatalogPython.htm __________________ All living things seek to control their own destiny....this is the purpose of life
	Register to Reply Score 1 for Good Answer

No more "Almost" Good Answers.

Bayes The Engineer Join Date: Feb 2005 Location: Albany, New York Posts: 5060 Good Answers: 129	#12 In reply to #10 Re: Python Resources on the Internet 07/14/2016 8:24 AM
	That second link is exactly the sort of thing I'm looking for, thanks!
	Register to Reply

phoenix911 Guru Join Date: Aug 2007 Posts: 23647 Good Answers: 420	#13 In reply to #12 Re: Python Resources on the Internet 07/14/2016 3:57 PM
	that is a good library. __________________ “ When people get what they want, they are often surprised when they get what they deserve " - James Wood
	Register to Reply

SolarEagle Guru Join Date: Mar 2007 Location: by the beach in Florida Posts: 33392 Good Answers: 1817	#14 In reply to #12 Re: Python Resources on the Internet 07/14/2016 3:58 PM
	Go in peace.... __________________ All living things seek to control their own destiny....this is the purpose of life
	Register to Reply

BruceFlorida Guru Join Date: Jul 2008 Posts: 1688 Good Answers: 145	#15 Re: Python Resources on the Internet 07/14/2016 4:48 PM
	Not what you are asking for but wanted to make sure you know about "wget". It has been many years since I have used python but if I wanted to get pages now I (currently using perl and php a lot) would use a perl system command to have the shell execute a "wget" to bring the web page into a local file. I would then process the file with text processing code rather than web page processing code. The command "wget" is probably not helpful to you since you got good links but it is a good item to have in your bag of tricks. __________________ Few things limit our potential as much as knowing answers and setting aside questions.
	Register to Reply

Bayes The Engineer Join Date: Feb 2005 Location: Albany, New York Posts: 5060 Good Answers: 129	#16 Re: Python Resources on the Internet 07/15/2016 7:38 AM
	Hi All, Appreciate the help, wanted to give an update. Came across this link https://first-web-scraper.readthedocs.io/en/latest/#act-3-web-scraping which has helped a lot. At this point I can go to a website and download the contents and even parse most of those contents (though right now I'm figuring out how to deal with these things /n/r/t/t (annoying). I'm pretty excited because once I've got those gone I'll have the data and I'll be able to save it to a .csv file which means I'll be good to go on doing some graphs. Some of the links you guys provided helped me understand terminology better which helped with my search plus gave me some insights on what modules I should be researching, so thanks for that.
	Register to Reply

"Almost" Good Answers:

Check out these comments that don't yet have enough votes to be "official" good answers and, if you agree with them, vote them!

#10 "Re: Python Resources on the Internet" by SolarEagle on 07/13/2016 12:36 PM (score 1)

Copy to Clipboard

Users who posted comments:

Bayes (7); BruceFlorida (1); Doggerel (1); JohnDG (4); phoenix911 (1); SolarEagle (2)

	Got Something to Say?
	Ask a Question Start a Discussion

	Search this Forum
	General Discussion: Go

CR4 Sections

	Aerospace
	BioMech & BioMed
	Chemical & Material Science
	Civil Engineering
	Communications & Electronics
	Education and Engineering Careers
	Electrical Engineering
	General Discussion
	Instrumentation
	Manufacturing
	Mechanical Engineering
	New Technologies & Research
	Software & Programming
	Sustainable Engineering
	Transportation

	Site Directories
	Forum Directory Blog Directory User Group Directory RSS Feeds CR4 Rules of Conduct CR4 FAQ Site Glossary Who's Online (78 right now) Show Members

Comments Format:

Subscribe to Discussion:

Rating Vote:

"Almost" Good Answers:

Re: Python Resources on the Internet

Re: Python Resources on the Internet

Re: Python Resources on the Internet

Re: Python Resources on the Internet

Re: Python Resources on the Internet

Re: Python Resources on the Internet

Re: Python Resources on the Internet

Re: Python Resources on the Internet

Re: Python Resources on the Internet

Re: Python Resources on the Internet

Re: Python Resources on the Internet

Re: Python Resources on the Internet

Re: Python Resources on the Internet

Re: Python Resources on the Internet

Re: Python Resources on the Internet

Re: Python Resources on the Internet

"Almost" Good Answers:

Users who posted comments:

Got Something to Say?

Search this Forum

CR4 Sections

Site Directories