Software

Ask Software Development Questions and Get Answers ASAP

Ask a Computer Expert,
Get an Answer ASAP!

Request: Given a URL for a currently live site (ex. -

Customer Question
http:\\www.origin.abc.com\abc.html), I want to crawl...
Request: Given a URL for a currently live site (ex. - http:\\www.origin.abc.com\abc.html), I want to crawl the entire site and download it to my local hard drive (html, images, etc.). This is typically relatively simple - as you know there are many tools to do this. The twist here is that all the links on teh live site are broken, and need to be adjusted for the site to be crawl-able. The adjustment is a very specific and simple one: ".origin" must go before the domain name. So in the above example, the site links all go to something like ex. - http:\\www.abc.com\xyz.html - which will always be broken. They will need a .origin added to work/for site to be automatically crawl-able - i.e. the above link would need to be turned into http:\\www.origin.abc.com\xyz.html.
Submitted: 6 months ago.Category: Software
Show More
Show Less
Ask Your Own Software Question
Customer reply replied 6 months ago
Example: https://www.origin.motorcraft.com/ar/sau/ is a web site I need to download. It's a relatively simple, content only page (save for dealer locator, which I don't need to download). But notice all the links on the page go to https://www.motorcraft.com - so a crawler will break unless it adds .origin before motorcraft.com for each link it attempts to crawl. So https://www.motorcraft.com would need to become https://www.origin.motorcraft.com.
Customer reply replied 6 months ago
Are you there? This is urgent and it's been about 1.5hrs. I have to get this going very soon.
Answered in 1 day by:
1/2/2018
Software technician: Evan Davis,
 replied 6 months ago
Evan Davis
Category: Software
Satisfied Customers: 18
Experience: Senior Applications Analyst at University of Kentucky HealthCare
Verified

If i'm understanding the question correctly, you need to change the links on a site to be prefixed with .origin?

Or is it that you are trying to get a crawler to prefix it?

One temp fix for this that may "Trick" the crawler into thinking it's looking in the right place is to add the URL's to the host file that point to the resolved URL.

For example, if the crawler is looking for origin.motorcraft.com you could setup host file entry for the following:

origin.motorcraft.com motorcraft.com

Let me know if i'm understanding correctly

Thanks much!

Ask Your Own Software Question
Customer reply replied 6 months ago
Thanks, ***** ***** the issue is it would need to be more like a wildcard replacement: Anytime you are crawling a page with a link to *w.motorcraft.*, instead go to $1w.origin.motorcraft.$2
Software technician: Evan Davis,
 replied 6 months ago

Thanks for the additional info, I think i'm with you but it may be beyond the scope of my expertise. It sounds more like a change in the method of crawling or the crawling application that is to be used. I'm going to put this back in the queue for another expert to pick up and assist.

Ask Your Own Software Question
Was this answer helpful?

How JustAnswer works

step-image
Describe your issueThe assistant will guide you
step-image
Chat 1:1 with a software technicianLicensed Experts are available 24/7
step-image
100% satisfaction guaranteeGet all the answers you need
Ask Jess M. Your Own Question
Jess M.
Jess M.
Jess M., Computer Support Specialist
Category: Software
Satisfied Customers: 775
775 Satisfied Customers
Experience: Computer Software Support specialist for more that 10 years

Jess M. is online now

A new question is answered every 9 seconds

How JustAnswer works:

  • Ask an ExpertExperts are full of valuable knowledge and are ready to help with any question. Credentials confirmed by a Fortune 500 verification firm.
  • Get a Professional AnswerVia email, text message, or notification as you wait on our site. Ask follow up questions if you need to.
  • 100% Satisfaction GuaranteeRate the answer you receive.

JustAnswer in the News:

Ask-a-doc Web sites: If you've got a quick question, you can try to get an answer from sites that say they have various specialists on hand to give quick answers... Justanswer.com.
JustAnswer.com...has seen a spike since October in legal questions from readers about layoffs, unemployment and severance.
Web sites like justanswer.com/legal
...leave nothing to chance.
Traffic on JustAnswer rose 14 percent...and had nearly 400,000 page views in 30 days...inquiries related to stress, high blood pressure, drinking and heart pain jumped 33 percent.
Tory Johnson, GMA Workplace Contributor, discusses work-from-home jobs, such as JustAnswer in which verified Experts answer people’s questions.
I will tell you that...the things you have to go through to be an Expert are quite rigorous.

What Customers are Saying:

My Expert answered my question promptly and he resolved the issue totally. This is a great service. I am so glad I found it I will definitely use the service again if needed.

One Happy CustomerNew York

Wonderful service, prompt, efficient, and accurate. Couldn't have asked for more. I cannot thank you enough for your help.

Mary C.Freshfield, Liverpool, UK

This expert is wonderful. They truly know what they are talking about, and they actually care about you. They really helped put my nerves at ease. Thank you so much!!!!

AlexLos Angeles, CA

Thank you for all your help. It is nice to know that this service is here for people like myself, who need answers fast and are not sure who to consult.

GPHesperia, CA

I couldn't be more satisfied! This is the site I will always come to when I need a second opinion.

JustinKernersville, NC

Just let me say that this encounter has been entirely professional and most helpful. I liked that I could ask additional questions and get answered in a very short turn around.

EstherWoodstock, NY

Thank you so much for taking your time and knowledge to support my concerns. Not only did you answer my questions, you even took it a step further with replying with more pertinent information I needed to know.

RobinElkton, Maryland

< Previous | Next >

Meet the Experts:

Jess M.

Jess M.

Computer Support Specialist

775 satisfied customers

Computer Software Support specialist for more that 10 years

Chris L.

Chris L.

Support Specialist

526 satisfied customers

Certified Software expert with over 10 years experience.

Jins M. N.

Jins M. N.

Computer Hardware Engineer

517 satisfied customers

12+ years experience. Expert in installing and problem fixing of softwares.

Kamil Anwar

Kamil Anwar

Software Specialist

514 satisfied customers

8+ Years of Experience. / CCNA (S), CCNA (W), CCNA (RS), MCTS, MBCs.

IT Miro

IT Miro

Computer Scientist

511 satisfied customers

Bachelor's Degree in Information Technology, Microsoft Certified Professional

Steve Herrod

Steve Herrod

Computer Support Specialist

220 satisfied customers

Familiar with a wide variety of software and experienced in user training/support

Byron

Byron

Computer Support Specialist

187 satisfied customers

12 years in operations at a large law firm supporting both retail and specialty applications.

< Previous | Next >

Related Software Questions
I need to do inline editing in datatables. I need a FREE
I need to do inline editing in datatables. I need a FREE solution. Please give me links to free code that does this. see https://editor.datatables.net/examples/inline-editing/index.htmlAlso, if there … read more
caraleneK
caraleneK
Service Manager
1,884 satisfied customers
SEO Tool to crawl websites and generate site maps. I need
SEO Tool to crawl websites and generate site maps. I need the tool to go at least 3 levels deep. PHP script is fine, online tool if free (or very inexpensive), chrome extension fine. Would be nice if … read more
GeekGal
GeekGal
IT Director
Bachelors Degree
562 satisfied customers
Need a tool to compare google search result links with
Need a tool to compare google search result links with website. I want to find all the links for a domain on google. For example, if I do site:www.foo.com I will get all the links in the google search… read more
Tim
Tim
Bachelors in Information Technology
12 satisfied customers
In google search pages I see URLs to a development site that
In google search pages I see URLs to a development site that I do NOT want indexed on search pages. I want to delete these URL's from the google search pages. When I own the property in google webmast… read more
Chris L.
Chris L.
Support Specialist
Vocational, Technical or Trade Scho
526 satisfied customers
Is there anything that you can submit multiple reports, of
Is there anything that you can submit multiple reports, of family records, each report with names, address, phone numbers, and relatives that will locate all the commons on these and identify?… read more
Russell H.
Russell H.
Computer Technology Consultant
Vocational, Technical or Trade Scho
789 satisfied customers
I need several options to stop or prevent unwanted visitors
I need several options to stop or prevent unwanted visitors to website. I think people are clicking on hyperlinks on hacked websites or computers and are landing on my website home page. This can hurt… read more
Jess M.
Jess M.
Computer Support Specialist
775 satisfied customers
I have a question about google. if there is a link that
I have a question about google. if there is a link that appears on a google page----that is linked to a newspaper & the newspaper takes down the link----how long does it take for google to remove it f… read more
Quiksilver07070
Quiksilver07070
Remote Technical Support
Associate Degree
68 satisfied customers
How do I get a report google search links are NOT valid
How do I get a report for all google search links are NOT valid for a website? For example, I want to find all the invalid google links for the site dublindeck.com When I put this "site:dublindeck.com… read more
Michael Norman
Michael Norman
Director of Technology
Master of Business Administration
78 satisfied customers
Our site isn't visible in google search. It used to be until
Our site isn't visible in google search. It used to be until an affiliated third party made a change to resolve a security issue. We did not authorize this and now they aren't able to tell us what cha… read more
Jess M.
Jess M.
Computer Support Specialist
775 satisfied customers
My name is ***** ***** when I was ill last week I must have
My name is ***** ***** when I was ill last week I must have put something on my facebook that was not nice. I was on medication and have not been into my account since then. I had a phone call from a … read more
Kamil Anwar
Kamil Anwar
Software Specialist
Bachelor's Degree
514 satisfied customers
For a wordpress site, I need to create a page that requires
For a wordpress site, I need to create a page that requires a sign in or a password ***** access. How do i do this? Example: The main site is for retail customers, but I need an informational page for… read more
Michael Hannigan
Michael Hannigan
Bachelor\u0027s Program
239 satisfied customers
***TheDoctor*** The domain, insideconnectionshome.com, does
***TheDoctor*** The domain, insideconnectionshome.com, does not work because my client's ex-partner dropped the ball, disappeared and did not pay the Yahoo! web hosting account for several months. I a… read more
Michael Hannigan
Michael Hannigan
Bachelor\u0027s Program
239 satisfied customers
on Google webmaster tools in manuel actions I got hit with
on Google webmaster tools in manuel actions I got hit with violation of Google's Webmaster Guidelines for cheapautoinsuranceco.com . Thin content with little or no added value This site appears to con… read more
MrBargain
MrBargain
Friendly Neighborhood Geek
Associate Degree
18 satisfied customers
Could you please remove the cached images of our house at 63
Could you please remove the cached images of our house atXXXXXDoonside NSW 2767 as it was sold last week. We have a criminal next door who has been harrassing us and we need to get the images off the … read more
Andy
Andy
Design Engineer
MS Electrical Engineering
5,360 satisfied customers
Is there software that will enable me to download to the hard
Is there software that will enable me to download to the hard drive Live stream videos from www.livestream.com? I have software to download youtube videos, but I want also to download livestream video… read more
James
James
Network & System Administrator
Bachelor\u0027s Degree
9,067 satisfied customers
This update has failed a number of times. What is wrong?Security
This update has failed a number of times. What is wrong? Security update for .NET Framework 2.0 SP2 och 3.5 SP1 i Windows Server 2003 and Windows XP x86 (KB2478658)… read more
Molinari
Molinari
Professional Tech Expert
CompTIA A+ Certification
4,157 satisfied customers
I am in the process of publishing two porn websites. One site
I am in the process of publishing two porn websites. One site is going to be a photo gallery the other site is going to be for videos. Each site is going to be a membership or password protected subsc… read more
Chris L.
Chris L.
Support Specialist
Vocational, Technical or Trade Scho
526 satisfied customers
WordPress expert with organic ranking knowledge. My websites
WordPress expert with organic ranking knowledge. My websites organically ranks one or 2 by the search engines web crawlers. my concern is this input by a wordpress expert. "By default, the main page o… read more
Jess M.
Jess M.
Computer Support Specialist
775 satisfied customers
Disclaimer: Information in questions, answers, and other posts on this site ("Posts") comes from individual users, not JustAnswer; JustAnswer is not responsible for Posts. Posts are for general information, are not intended to substitute for informed professional advice (medical, legal, veterinary, financial, etc.), or to establish a professional-client relationship. The site and services are provided "as is" with no warranty or representations by JustAnswer regarding the qualifications of Experts. To see what credentials have been verified by a third-party service, please click on the "Verified" symbol in some Experts' profiles. JustAnswer is not intended or designed for EMERGENCY questions which should be directed immediately by telephone or in-person to qualified professionals.

Disclaimer: Information in questions, answers, and other posts on this site ("Posts") comes from individual users, not JustAnswer; JustAnswer is not responsible for Posts. Posts are for general information, are not intended to substitute for informed professional advice (medical, legal, veterinary, financial, etc.), or to establish a professional-client relationship. The site and services are provided "as is" with no warranty or representations by JustAnswer regarding the qualifications of Experts. To see what credentials have been verified by a third-party service, please click on the "Verified" symbol in some Experts' profiles. JustAnswer is not intended or designed for EMERGENCY questions which should be directed immediately by telephone or in-person to qualified professionals.

Show MoreShow Less

Ask Your Question

x