Validate External Links - Revision history

Iritscen: updated link for websvn browser's new URL format

2025-03-07T16:49:17Z

updated link for websvn browser's new URL format

← Older revision		Revision as of 16:49, 7 March 2025
Line 7:		Line 7:

	==Running and contributing==		==Running and contributing==
	The project is found [https://websvn.illy.bz/~~listing.php?repname=~~Oni2~~&path=~~%~~2FValidate+External+Links~~%2F HERE]. Along with the Bash shell script itself, you'll find documentation on how to run ValExtLinks on your own computer as well as resources for contributing to the code.		The project is found [https://websvn.illy.bz/browse/Oni2/Validate%20External%20Links/ HERE]. Along with the Bash shell script itself, you'll find documentation on how to run ValExtLinks on your own computer as well as resources for contributing to the code.

	==Fixing link issues in a report==		==Fixing link issues in a report==

Iritscen: link fix

2024-03-24T14:42:29Z

link fix

← Older revision		Revision as of 14:42, 24 March 2024
Line 11:		Line 11:
	==Fixing link issues in a report==		==Fixing link issues in a report==
	Here are the codes that you'll see applied to problem links in the report.		Here are the codes that you'll see applied to problem links in the report.
	*'''NG''': In most cases, fixing an NG ("no good") link will mean finding the desired web page in the Internet Archive's [https://archive.org~~/web~~/ Wayback Machine] and linking to that archived copy instead. In some cases, an NG link will not be salvageable and should be either removed from the page or, if the link was a part of a conversation and it would be confusing for it to be absent, it should be surrounded in <code>nowiki</code> tags [[Special:Diff/40524\|like this]] to prevent it from showing up in future reports.		*'''NG''': In most cases, fixing an NG ("no good") link will mean finding the desired web page in the Internet Archive's [https://web.archive.org/ Wayback Machine] and linking to that archived copy instead. In some cases, an NG link will not be salvageable and should be either removed from the page or, if the link was a part of a conversation and it would be confusing for it to be absent, it should be surrounded in <code>nowiki</code> tags [[Special:Diff/40524\|like this]] to prevent it from showing up in future reports.
	**Val automatically queries the Archive for the latest snapshot of each NG page and will put this snapshot URL in its report. Note that you should still check this snapshot to make sure it has the desired content. You may have to go further back in the Wayback Machine to find the proper snapshot to use. Sometimes the Archive simply never got around to archiving a given site. In that case, you will need to follow the advice above as to deleting the link or marking it with <code>nowiki</code> tags.		**Val automatically queries the Archive for the latest snapshot of each NG page and will put this snapshot URL in its report. Note that you should still check this snapshot to make sure it has the desired content. You may have to go further back in the Wayback Machine to find the proper snapshot to use. Sometimes the Archive simply never got around to archiving a given site. In that case, you will need to follow the advice above as to deleting the link or marking it with <code>nowiki</code> tags.
	**Note: In a typical run of ValExtLinks across the 3,500 links on the wiki, 1-3 sites will happen to be offline at the moment or the HTTP packets requesting them will get lost in cyberspace. Before attempting to perform any of the above fixes, try the link manually to make sure the site is really down. Even if it is, you might want to wait for next week's Val report to see if the site is permanently dead.		**Note: In a typical run of ValExtLinks across the 3,500 links on the wiki, 1-3 sites will happen to be offline at the moment or the HTTP packets requesting them will get lost in cyberspace. Before attempting to perform any of the above fixes, try the link manually to make sure the site is really down. Even if it is, you might want to wait for next week's Val report to see if the site is permanently dead.

Iritscen: link fix

2023-12-03T17:30:07Z

link fix

← Older revision		Revision as of 17:30, 3 December 2023
Line 7:		Line 7:

	==Running and contributing==		==Running and contributing==
	The project is found [~~http~~://websvn.~~chrilly~~.~~net~~/listing.php?repname=Oni2&path=%2FValidate+External+Links%2F HERE]. Along with the Bash shell script itself, you'll find documentation on how to run ValExtLinks on your own computer as well as resources for contributing to the code.		The project is found [https://websvn.illy.bz/listing.php?repname=Oni2&path=%2FValidate+External+Links%2F HERE]. Along with the Bash shell script itself, you'll find documentation on how to run ValExtLinks on your own computer as well as resources for contributing to the code.

	==Fixing link issues in a report==		==Fixing link issues in a report==

Iritscen: wording

2023-05-21T19:57:05Z

wording

← Older revision		Revision as of 19:57, 21 May 2023
Line 2:		Line 2:

	==Background==		==Background==
	While MediaWiki makes it easy to find bad intrawiki links (links to nonexistent pages on our own wiki), marking them in red and providing tools like [[Special:Wantedpages]], there is no automatic check of external (outbound) links. MediaWiki compiles external links into a table, but it does not ping the URLs to see if they give any response. Over the years, many links on our wiki went dead as the Web changed and various file hosts went out of business. ValExtLinks has been used to fix ~~over 1,000~~ link issues on OniGalore such as 404s and redirects.		While MediaWiki makes it easy to find bad intrawiki links (links to nonexistent pages on our own wiki) by marking them in red and providing tools like [[Special:Wantedpages]], there is no automatic check of external (outbound) links. MediaWiki compiles external links into a table, but it does not ping the URLs to see if they give any response. Over the years, many links on our wiki went dead as the Web changed and various file hosts went out of business. ValExtLinks has been used to fix thousands of link issues on OniGalore such as 404s and redirects.

	Here's how the process works: at 6:20am and 2:20pm (GMT) ~~each day~~, a script written by [[User:Admin\|Alloc]] dumps the wiki's external links table to [https://wiki.oni2.net/w/extlinks.csv this location]. ValExtLinks, which Iritscen runs on his computer periodically, walks through the exported table and looks for URLs that return problematic ~~codes~~ such as 404. It also detects other lesser problems with links. Val then makes suggestions for fixing these links and uploads its ~~report~~ in HTML, RTF and TXT formats to [http://iritscen.oni2.net/val/ this directory]. A wiki editor can then review the ~~report~~ and act accordingly.		Here's how the process works: twice a day (6:20am and 2:20pm GMT), a script written by [[User:Admin\|Alloc]] dumps the wiki's external links table to [https://wiki.oni2.net/w/extlinks.csv this location]. ValExtLinks, which Iritscen runs on his computer periodically, walks through the exported table and looks for URLs that return problematic responses such as 301 and 404. It also detects other lesser problems with links. Val then makes suggestions for fixing these links and uploads its findings in HTML, RTF and TXT formats to [http://iritscen.oni2.net/val/ this directory]. Any wiki editor can then review the reports and act accordingly.

	==~~How~~ to ~~fix~~ link issues==		==Running and contributing==
	Here are the codes that you'll see on problem links in the report.		The project is found [http://websvn.chrilly.net/listing.php?repname=Oni2&path=%2FValidate+External+Links%2F HERE]. Along with the Bash shell script itself, you'll find documentation on how to run ValExtLinks on your own computer as well as resources for contributing to the code.
	*'''NG''': In most cases, fixing an NG ("no good") link will mean finding the desired web page in the Internet Archive's [https://archive.org/web/ Wayback Machine] and linking to that archived ~~page~~ instead. In some cases, an NG link will not be salvageable and should be either removed from the page or, if the link was a part of a conversation and it would be confusing for it to be absent, it should be surrounded in nowiki tags [[Special:Diff/~~16377/26212~~\|like this]] to prevent it from showing up in future reports.
	**Val automatically queries the Archive for the latest snapshot of each NG page and will put ~~the returned~~ snapshot URL in its report. Note that you still ~~have~~ to ~~verify this link by clicking on~~ it~~, as it may not have~~ the ~~correct~~ content. You may have to go further back in the Wayback Machine to find the proper snapshot to use. Sometimes the Archive simply never got around to archiving a given site. In that case, you will need to follow the advice above as to deleting the link or marking it with nowiki tags.		==Fixing link issues in a report==
	**Note: In a typical run of ~~Val~~ across the 3,~~000+~~ links on the wiki, 1-3 sites will happen to be offline at the moment or the HTTP packets requesting them will get lost in the ~~Internet~~. ~~It's best~~ to wait for ~~another~~ Val report to ~~make sure that~~ the ~~URL~~ is ~~really~~ dead ~~before performing any of the above fixes~~.		Here are the codes that you'll see applied to problem links in the report.
	*'''RD''': The site is redirecting the browser to a new page. ~~The~~ new page ~~should be evaluated, and if it~~ has the content we intended to link to ~~then~~ we should update the link to point to ~~the~~ new location. ~~However, many~~ redirects actually ~~are~~ "soft 404s" ~~and simply redirect~~ the browser to the site's main page. In this case, an RD link needs to be treated like an NG link (see above).		*'''NG''': In most cases, fixing an NG ("no good") link will mean finding the desired web page in the Internet Archive's [https://archive.org/web/ Wayback Machine] and linking to that archived copy instead. In some cases, an NG link will not be salvageable and should be either removed from the page or, if the link was a part of a conversation and it would be confusing for it to be absent, it should be surrounded in <code>nowiki</code> tags [[Special:Diff/40524\|like this]] to prevent it from showing up in future reports.
	*'''EI''': An "external internal" link, that is, a full URL for a page that is on our own wiki and which should simply be an [[Help:Editing#~~Intrawiki_links~~\|intrawiki link]]. Sometimes an "external internal" may seem to be necessary but can be avoided with one of these special wiki features:		**Val automatically queries the Archive for the latest snapshot of each NG page and will put this snapshot URL in its report. Note that you should still check this snapshot to make sure it has the desired content. You may have to go further back in the Wayback Machine to find the proper snapshot to use. Sometimes the Archive simply never got around to archiving a given site. In that case, you will need to follow the advice above as to deleting the link or marking it with <code>nowiki</code> tags.
	**If you want to link to a specific revision of a page, you might think you need a full URL [https://wiki.oni2.net/w/index.php?title=Oni&oldid=7685 like this one]. There's actually no need to link to any page at all, as the "ID" of a page revision ~~like the one~~ you see in that sample URL is unique wiki-wide. All you need to do is supply the revision ID to the Special:Permalink page like this — [[Special:Permalink/7685]] — and you're done.		**Note: In a typical run of ValExtLinks across the 3,500 links on the wiki, 1-3 sites will happen to be offline at the moment or the HTTP packets requesting them will get lost in cyberspace. Before attempting to perform any of the above fixes, try the link manually to make sure the site is really down. Even if it is, you might want to wait for next week's Val report to see if the site is permanently dead.
	**If you need to link to a diff between two revisions of a page, or between two different pages, plug the old and new revision numbers into the Special:Diff page like this: [[Special:Diff/21491/21492]] ~~(no need for page names, as explained above)~~.		*'''RD''': The site is redirecting the browser to a new page. If the new page has the content we intended to link to, we should update the link to point to this new location. Be aware that some redirects are actually "soft 404s", redirecting the browser to the site's main page. In this case, an RD link needs to be treated like an NG link (see above).
	**If there's no way around a bare URL, see "Exceptions" below to remove the link from the report.		*'''EI''': An "external internal" link, that is, a full URL for a page that is on our own wiki and which should simply be an [[Help:Editing#Intrawiki links\|intrawiki link]]. Sometimes an "external internal" may seem to be necessary but can be avoided with one of these special wiki features:
	*'''IW''': This marks an external link (bare URL) to another wiki which could be an [[Help:Editing#~~Interwiki_links~~\|interwiki link]]. Interwiki links are shorter and more resistant to rot. The suggested interwiki link markup will be given in the report. For foreign-language Wikipedia pages, you can add a language code, e.g. <nowiki>[[wp:de:Test]]</nowiki> for the German version of the page.		**If you want to link to a specific revision of a page, you might think you need a full URL [https://wiki.oni2.net/w/index.php?title=Oni&oldid=7685 like this one]. There's actually no need to link to any page at all, as the "ID" of a page revision (which you will see in that sample URL) is unique wiki-wide. All you need to do is supply the revision ID to the Special:Permalink page like this — [[Special:Permalink/7685]] — and you're done.
	*'''(xxx)''': The HTTP response code (see reference [[/HTTP codes\|HERE]]).		**If you want to link to a specific revision as a diff from the previous revision of that page, plug the revision number into the Special:Diff page like this: [[Special:Diff/40550]] (no need for page names, as explained above). To link to a diff between two non-contiguous revisions of a page or between two different pages, plug the old and new revision numbers into the Special:Diff page like this: [[Special:Diff/21491/21492]].
	*'''(000-xx)''': The ~~response~~ code from the Unix tool 'curl', for cases where it ~~did not~~ get an HTTP response code~~. See~~ [[/Curl codes\|HERE]] ~~for a list of codes~~. The most common curl error by far is "000-28", a timeout from an unresponsive site.		**If there's no way around a bare URL, see the "Exceptions" section below to learn how to remove the link from the report.
			*'''IW''': This marks an external link (bare URL) to another wiki which could be an [[Help:Editing#Interwiki links\|interwiki link]]. Interwiki links are shorter and more resistant to rot. The suggested interwiki link markup will be given in the report. For foreign-language Wikipedia pages, you can add a language code, e.g. <nowiki>[[wp:de:Test]]</nowiki> for the German version of the page "Test".
			*'''(xxx)''': The HTTP response code (see HTTP code reference [[/HTTP codes\|HERE]]).
			*'''(000-xx)''': The exit code from the Unix tool 'curl', for cases where it failed to get an HTTP response code (see 'curl' code reference [[/Curl codes\|HERE]]). The most common 'curl' error by far is "000-28", a timeout from an unresponsive site.

	===Exceptions===		===Exceptions===
	Some links simply must be presented in an unconventional way which Val thinks is a problem. Some links return error codes but actually work fine. Such links can be added to the [[/Exceptions\|exceptions list]] in order to hide them in future reports.		Some links simply must be presented in an unconventional way which Val thinks is a problem. Some links return error codes but actually work fine. Such links can be added to the [[/Exceptions\|exceptions list]] in order to hide them in future reports.

	In the summary at the bottom of ~~the~~ report, Val will list any exception that didn't have the intended effect because the link is no longer present on the listed page, or because it doesn't return ~~that~~ error code anymore. You can then edit the ~~above~~ exceptions list accordingly. Note that the HTML report only gives the number of issues detected, and the list of issues is found in the RTF and TXT versions of the report.		In the summary at the bottom of a ValExtLinks report, Val will list any exception that didn't have the intended effect because the link is no longer present on the listed page or because it doesn't return the expected error code anymore. You can then edit the wiki's exceptions list accordingly. Note that the HTML report only gives the number of exception issues detected, and the actual list of issues is found in the RTF and TXT versions of the report.

	~~==Source code==~~
	The project is found [http://websvn.chrilly.net/listing.php?repname=Oni2&path=%2FValidate+External+Links%2F HERE]. Along with the Bash script itself, you'll find documentation on how to run ValExtLinks on your own computer as well as resources for contributing to the code.

	[[Category:Wiki Support]]		[[Category:Wiki Support]]

Iritscen: this feature already exists, I just haven't been using it

2023-05-07T00:51:35Z

this feature already exists, I just haven't been using it

← Older revision		Revision as of 00:51, 7 May 2023
Line 4:		Line 4:
	While MediaWiki makes it easy to find bad intrawiki links (links to nonexistent pages on our own wiki), marking them in red and providing tools like [[Special:Wantedpages]], there is no automatic check of external (outbound) links. MediaWiki compiles external links into a table, but it does not ping the URLs to see if they give any response. Over the years, many links on our wiki went dead as the Web changed and various file hosts went out of business. ValExtLinks has been used to fix over 1,000 link issues on OniGalore such as 404s and redirects.		While MediaWiki makes it easy to find bad intrawiki links (links to nonexistent pages on our own wiki), marking them in red and providing tools like [[Special:Wantedpages]], there is no automatic check of external (outbound) links. MediaWiki compiles external links into a table, but it does not ping the URLs to see if they give any response. Over the years, many links on our wiki went dead as the Web changed and various file hosts went out of business. ValExtLinks has been used to fix over 1,000 link issues on OniGalore such as 404s and redirects.

	Here's how the process works: at 6:20am and 2:20pm (GMT) each day, a script written by [[User:Admin\|Alloc]] dumps the wiki's external links table to [https://wiki.oni2.net/w/extlinks.csv this location]. ValExtLinks, which Iritscen runs on his computer periodically, walks through the exported table and looks for URLs that return problematic codes such as 404. It also detects other lesser problems with links. Val then makes suggestions for fixing these links and uploads its report in HTML, RTF and TXT formats to [http://iritscen.oni2.net/val/ ~~THIS~~] ~~directory~~. A wiki editor can then review the report and act accordingly.		Here's how the process works: at 6:20am and 2:20pm (GMT) each day, a script written by [[User:Admin\|Alloc]] dumps the wiki's external links table to [https://wiki.oni2.net/w/extlinks.csv this location]. ValExtLinks, which Iritscen runs on his computer periodically, walks through the exported table and looks for URLs that return problematic codes such as 404. It also detects other lesser problems with links. Val then makes suggestions for fixing these links and uploads its report in HTML, RTF and TXT formats to [http://iritscen.oni2.net/val/ this directory]. A wiki editor can then review the report and act accordingly.

	==How to fix link issues==		==How to fix link issues==
	Here are the codes that you'll see on problem links in the report.		Here are the codes that you'll see on problem links in the report.
	*'''NG''': In most cases, fixing an ~~[[wp:List_of_Japanese_Latin_alphabetic_abbreviations#N\|~~"NG"]] link will mean finding the desired web page in the Internet Archive's [https://archive.org/web/ Wayback Machine] and linking to that archived page instead. In some cases, an NG link will not be salvageable and should be either removed from the page or, if the link was a part of a conversation and it would be confusing for it to be absent, it should be surrounded in nowiki tags [[Special:Diff/16377/26212\|like this]] to prevent it from showing up in future reports.		*'''NG''': In most cases, fixing an NG ("no good") link will mean finding the desired web page in the Internet Archive's [https://archive.org/web/ Wayback Machine] and linking to that archived page instead. In some cases, an NG link will not be salvageable and should be either removed from the page or, if the link was a part of a conversation and it would be confusing for it to be absent, it should be surrounded in nowiki tags [[Special:Diff/16377/26212\|like this]] to prevent it from showing up in future reports.
	**Val automatically queries the Archive for the latest snapshot of each NG page and will put the returned snapshot URL in its report. Note that you still have to verify this link by clicking on it, as it may not have the correct content. You may have to go further back in the Wayback Machine to find the proper snapshot to use. Sometimes the Archive simply never got around to archiving a given site. In that case, you will need to follow the advice above as to deleting the link or marking it with nowiki tags.		**Val automatically queries the Archive for the latest snapshot of each NG page and will put the returned snapshot URL in its report. Note that you still have to verify this link by clicking on it, as it may not have the correct content. You may have to go further back in the Wayback Machine to find the proper snapshot to use. Sometimes the Archive simply never got around to archiving a given site. In that case, you will need to follow the advice above as to deleting the link or marking it with nowiki tags.
	**Note: In a typical run of Val across the 3,000+ links on the wiki, 1-3 sites will happen to be offline at the moment or the HTTP packets requesting them will get lost in the Internet. It's best to wait for another Val report to make sure that the URL is really dead before performing any of the above fixes.		**Note: In a typical run of Val across the 3,000+ links on the wiki, 1-3 sites will happen to be offline at the moment or the HTTP packets requesting them will get lost in the Internet. It's best to wait for another Val report to make sure that the URL is really dead before performing any of the above fixes.
Line 24:		Line 24:

	In the summary at the bottom of the report, Val will list any exception that didn't have the intended effect because the link is no longer present on the listed page, or because it doesn't return that error code anymore. You can then edit the above exceptions list accordingly. Note that the HTML report only gives the number of issues detected, and the list of issues is found in the RTF and TXT versions of the report.		In the summary at the bottom of the report, Val will list any exception that didn't have the intended effect because the link is no longer present on the listed page, or because it doesn't return that error code anymore. You can then edit the above exceptions list accordingly. Note that the HTML report only gives the number of issues detected, and the list of issues is found in the RTF and TXT versions of the report.

	~~==Coming features==~~
	*It seems that many external links no longer display the content that they were intended to display. In many cases, web sites are silently redirecting the user to their main page without using the appropriate code that indicates the content was not found. Only visual inspections of the pages can catch these issues. Once we have dealt with the more obvious link issues, the screenshot feature in the script can be activated, which will allow us to confirm if the "OK" links are loading the intended page.

	==Source code==		==Source code==

Iritscen: /* Exceptions */ about the audit

2022-08-23T14:22:27Z

Exceptions: about the audit

← Older revision		Revision as of 14:22, 23 August 2022
Line 22:		Line 22:
	===Exceptions===		===Exceptions===
	Some links simply must be presented in an unconventional way which Val thinks is a problem. Some links return error codes but actually work fine. Such links can be added to the [[/Exceptions\|exceptions list]] in order to hide them in future reports.		Some links simply must be presented in an unconventional way which Val thinks is a problem. Some links return error codes but actually work fine. Such links can be added to the [[/Exceptions\|exceptions list]] in order to hide them in future reports.

			In the summary at the bottom of the report, Val will list any exception that didn't have the intended effect because the link is no longer present on the listed page, or because it doesn't return that error code anymore. You can then edit the above exceptions list accordingly. Note that the HTML report only gives the number of issues detected, and the list of issues is found in the RTF and TXT versions of the report.

	==Coming features==		==Coming features==

Iritscen: /* Coming features */ exceptions audit has been added

2022-08-23T14:18:47Z

Coming features: exceptions audit has been added

← Older revision		Revision as of 14:18, 23 August 2022
Line 25:		Line 25:
	==Coming features==		==Coming features==
	*It seems that many external links no longer display the content that they were intended to display. In many cases, web sites are silently redirecting the user to their main page without using the appropriate code that indicates the content was not found. Only visual inspections of the pages can catch these issues. Once we have dealt with the more obvious link issues, the screenshot feature in the script can be activated, which will allow us to confirm if the "OK" links are loading the intended page.		*It seems that many external links no longer display the content that they were intended to display. In many cases, web sites are silently redirecting the user to their main page without using the appropriate code that indicates the content was not found. Only visual inspections of the pages can catch these issues. Once we have dealt with the more obvious link issues, the screenshot feature in the script can be activated, which will allow us to confirm if the "OK" links are loading the intended page.

	*Over time, the exceptions list should be audited so it doesn't acquire [[wikt:cruft\|cruft]]. Val needs to report URLs in the list that aren't even in the external links table anymore, as well as URLs in the exceptions list which no longer return an error code.

	==Source code==		==Source code==

Iritscen: lots of wording improvements

2021-11-18T02:09:43Z

lots of wording improvements

Iritscen: /* How to fix link issues */ noting that at any given time, someone's web site is having a bad day

2021-05-16T15:06:47Z

How to fix link issues: noting that at any given time, someone's web site is having a bad day

← Older revision		Revision as of 15:06, 16 May 2021
Line 8:		Line 8:
	==How to fix link issues==		==How to fix link issues==
	Here are the codes that you'll see on problem links in the report.		Here are the codes that you'll see on problem links in the report.
	*'''NG''': In most cases, fixing an [[wp:List_of_Japanese_Latin_alphabetic_abbreviations#N\|"NG"]] link will mean finding the desired web page in the Internet Archive's [https://archive.org/web/ Wayback Machine] and linking to that archived page instead. In some cases, an NG link will not be ~~recoverable~~ and should be either removed from the page or, if the link was a part of a conversation and it would be confusing for it to be absent, it should be surrounded in nowiki tags [[Special:Diff/16377/26212\|like this]] to prevent it from showing up in future reports.		*'''NG''': In most cases, fixing an [[wp:List_of_Japanese_Latin_alphabetic_abbreviations#N\|"NG"]] link will mean finding the desired web page in the Internet Archive's [https://archive.org/web/ Wayback Machine] and linking to that archived page instead. In some cases, an NG link will not be salvageable and should be either removed from the page or, if the link was a part of a conversation and it would be confusing for it to be absent, it should be surrounded in nowiki tags [[Special:Diff/16377/26212\|like this]] to prevent it from showing up in future reports.
	**Val automatically queries the Archive for the latest snapshot of each NG page and will put the returned snapshot URL in its report. Note that you still have to verify this link by clicking on it, as it may not have the correct content. You may have to go further back in the Wayback Machine to find the proper snapshot to use. Sometimes the Archive simply never got around to archiving a given site. In that case, you will need to follow the advice above as to deleting the link or marking it with nowiki tags.		**Val automatically queries the Archive for the latest snapshot of each NG page and will put the returned snapshot URL in its report. Note that you still have to verify this link by clicking on it, as it may not have the correct content. You may have to go further back in the Wayback Machine to find the proper snapshot to use. Sometimes the Archive simply never got around to archiving a given site. In that case, you will need to follow the advice above as to deleting the link or marking it with nowiki tags.
			**Note: In a typical weekly run of Val across the 3,000+ links on the wiki, 1-3 sites will simply happen to be offline at the moment. It's best to wait a week for another Val report to make sure that the URL is really dead before performing any of the above fixes.
	*'''RD''': The site is redirecting the browser to a new page. The new page should be evaluated, and if it has the content we intended to link to then we should update the link to point to the new location. However, many redirects actually are "soft 404s" and simply redirect the browser to the site's main page. In this case, an RD link needs to be treated like an NG link (see above).		*'''RD''': The site is redirecting the browser to a new page. The new page should be evaluated, and if it has the content we intended to link to then we should update the link to point to the new location. However, many redirects actually are "soft 404s" and simply redirect the browser to the site's main page. In this case, an RD link needs to be treated like an NG link (see above).
	*'''EI''': An external link (bare URL) for a page on our own wiki that could simply be an [[Help:Editing#Intrawiki_links\|intrawiki link]]. Sometimes an "external internal" may seem to be necessary, but there's a special wiki feature that allows you to avoid it:		*'''EI''': An external link (bare URL) for a page on our own wiki that could simply be an [[Help:Editing#Intrawiki_links\|intrawiki link]]. Sometimes an "external internal" may seem to be necessary, but there's a special wiki feature that allows you to avoid it:

Iritscen: linking to ValBot

2021-04-05T16:37:08Z

linking to ValBot

← Older revision		Revision as of 16:37, 5 April 2021
Line 1:		Line 1:
	Developed by [[User:Iritscen\|Iritscen]], Validate External Links ("ValExtLinks" for short, or "Val" for even shorter) is a Bash shell script made to help fight the battle against [[wikt:linkrot\|link rot]] on OniGalore. The latest report on link issues is found '''[http://iritscen.oni2.net/val/ValExtLinks%20report.htm HERE]'''.		Developed by [[User:Iritscen\|Iritscen]], Validate External Links ("ValExtLinks" for short, or "Val" for even shorter) is a Bash shell script made to help fight the battle against [[wikt:linkrot\|link rot]] on OniGalore. The latest report on link issues is found '''[http://iritscen.oni2.net/val/ValExtLinks%20report.htm HERE]'''. Further link analysis which requires a bot is performed by [[User:ValBot\|ValBot]].

	==Background==		==Background==

← Older revision		Revision as of 02:09, 18 November 2021
Line 1:		Line 1:
	Developed by [[User:Iritscen\|Iritscen]], Validate External Links ("ValExtLinks" for short, or "Val" for even shorter) is a Bash shell script made to help fight the battle against [[wikt:linkrot\|link rot]] on OniGalore. The latest report on link issues is found '''[http://iritscen.oni2.net/val/ValExtLinks%20report.htm HERE]'''. Further link ~~analysis~~ which requires a bot is performed by [[User:ValBot\|ValBot]].		Developed by [[User:Iritscen\|Iritscen]], Validate External Links ("ValExtLinks" for short, or "Val" for even shorter) is a Bash shell script made to help fight the battle against [[wikt:linkrot\|link rot]] on OniGalore. The latest report on link issues is found '''[http://iritscen.oni2.net/val/ValExtLinks%20report.htm HERE]'''. Further link work which requires a bot is performed by [[User:ValBot\|ValBot]].

	==Background==		==Background==
	While MediaWiki makes it easy to find bad links to pages on our own wiki, marking them in red and providing tools like [[Special:~~Wantedpages\|~~Wantedpages]], there is no automatic check of external links. MediaWiki compiles external links into a table, but it does not ping the URLs to see if they give any response. Over the years, many links on our wiki went dead as the Web changed and various file hosts went out of business. ValExtLinks has been used to fix over 1,000 link issues on OniGalore.		While MediaWiki makes it easy to find bad intrawiki links (links to nonexistent pages on our own wiki), marking them in red and providing tools like [[Special:Wantedpages]], there is no automatic check of external (outbound) links. MediaWiki compiles external links into a table, but it does not ping the URLs to see if they give any response. Over the years, many links on our wiki went dead as the Web changed and various file hosts went out of business. ValExtLinks has been used to fix over 1,000 link issues on OniGalore such as 404s and redirects.

	Here's how the process works: at 6:20am and 2:20pm (GMT) each day, a script written by [[User:Admin\|Alloc]] dumps the wiki's external links table to [https://wiki.oni2.net/w/extlinks.csv this location]. ~~Val~~, which Iritscen runs on his computer periodically, walks through the exported table and looks for URLs that return problematic codes such as 404. It also detects other lesser problems with links. Val then makes suggestions for fixing these links and uploads its report ~~to the location above (note that an~~ RTF and TXT ~~version are also uploaded)~~. A wiki editor ~~reviews~~ the report and ~~acts~~ accordingly.		Here's how the process works: at 6:20am and 2:20pm (GMT) each day, a script written by [[User:Admin\|Alloc]] dumps the wiki's external links table to [https://wiki.oni2.net/w/extlinks.csv this location]. ValExtLinks, which Iritscen runs on his computer periodically, walks through the exported table and looks for URLs that return problematic codes such as 404. It also detects other lesser problems with links. Val then makes suggestions for fixing these links and uploads its report in HTML, RTF and TXT formats to [http://iritscen.oni2.net/val/ THIS] directory. A wiki editor can then review the report and act accordingly.

	==How to fix link issues==		==How to fix link issues==
Line 10:		Line 10:
	*'''NG''': In most cases, fixing an [[wp:List_of_Japanese_Latin_alphabetic_abbreviations#N\|"NG"]] link will mean finding the desired web page in the Internet Archive's [https://archive.org/web/ Wayback Machine] and linking to that archived page instead. In some cases, an NG link will not be salvageable and should be either removed from the page or, if the link was a part of a conversation and it would be confusing for it to be absent, it should be surrounded in nowiki tags [[Special:Diff/16377/26212\|like this]] to prevent it from showing up in future reports.		*'''NG''': In most cases, fixing an [[wp:List_of_Japanese_Latin_alphabetic_abbreviations#N\|"NG"]] link will mean finding the desired web page in the Internet Archive's [https://archive.org/web/ Wayback Machine] and linking to that archived page instead. In some cases, an NG link will not be salvageable and should be either removed from the page or, if the link was a part of a conversation and it would be confusing for it to be absent, it should be surrounded in nowiki tags [[Special:Diff/16377/26212\|like this]] to prevent it from showing up in future reports.
	**Val automatically queries the Archive for the latest snapshot of each NG page and will put the returned snapshot URL in its report. Note that you still have to verify this link by clicking on it, as it may not have the correct content. You may have to go further back in the Wayback Machine to find the proper snapshot to use. Sometimes the Archive simply never got around to archiving a given site. In that case, you will need to follow the advice above as to deleting the link or marking it with nowiki tags.		**Val automatically queries the Archive for the latest snapshot of each NG page and will put the returned snapshot URL in its report. Note that you still have to verify this link by clicking on it, as it may not have the correct content. You may have to go further back in the Wayback Machine to find the proper snapshot to use. Sometimes the Archive simply never got around to archiving a given site. In that case, you will need to follow the advice above as to deleting the link or marking it with nowiki tags.
	**Note: In a typical ~~weekly~~ run of Val across the 3,000+ links on the wiki, 1-3 sites will ~~simply~~ happen to be offline at the moment. It's best to wait ~~a week~~ for another Val report to make sure that the URL is really dead before performing any of the above fixes.		**Note: In a typical run of Val across the 3,000+ links on the wiki, 1-3 sites will happen to be offline at the moment or the HTTP packets requesting them will get lost in the Internet. It's best to wait for another Val report to make sure that the URL is really dead before performing any of the above fixes.
	*'''RD''': The site is redirecting the browser to a new page. The new page should be evaluated, and if it has the content we intended to link to then we should update the link to point to the new location. However, many redirects actually are "soft 404s" and simply redirect the browser to the site's main page. In this case, an RD link needs to be treated like an NG link (see above).		*'''RD''': The site is redirecting the browser to a new page. The new page should be evaluated, and if it has the content we intended to link to then we should update the link to point to the new location. However, many redirects actually are "soft 404s" and simply redirect the browser to the site's main page. In this case, an RD link needs to be treated like an NG link (see above).
	*'''EI''': An external link ~~(bare~~ URL) for a page on our own wiki ~~that could~~ simply be an [[Help:Editing#Intrawiki_links\|intrawiki link]]. Sometimes an "external internal" may seem to be necessary, but ~~there's a~~ special wiki ~~feature that allows you to avoid it~~:		*'''EI''': An "external internal" link, that is, a full URL for a page that is on our own wiki and which should simply be an [[Help:Editing#Intrawiki_links\|intrawiki link]]. Sometimes an "external internal" may seem to be necessary but can be avoided with one of these special wiki features:
	**If you want to link to a specific ~~version~~ of a page, ~~which used to require putting the~~ full URL, [https://wiki.oni2.net/w/index.php?title=Oni&oldid=7685 like this]. ~~In fact, there~~'s no need to link to any page at all, as the "ID" of ~~an edit,~~ like the one you see in that sample URL, is unique wiki-wide. All you need to do is supply the revision ID to the Special:Permalink page like this — [[Special:Permalink/7685]] — and you're done.		**If you want to link to a specific revision of a page, you might think you need a full URL [https://wiki.oni2.net/w/index.php?title=Oni&oldid=7685 like this one]. There's actually no need to link to any page at all, as the "ID" of a page revision like the one you see in that sample URL is unique wiki-wide. All you need to do is supply the revision ID to the Special:Permalink page like this — [[Special:Permalink/7685]] — and you're done.
	**If you need to link to a diff between two revisions of a page, or between two different pages, plug the old and new revision numbers into the Special:Diff page like this: [[Special:Diff/21491/21492]] (no need for page names, as explained above).		**If you need to link to a diff between two revisions of a page, or between two different pages, plug the old and new revision numbers into the Special:Diff page like this: [[Special:Diff/21491/21492]] (no need for page names, as explained above).
	**If there's no ~~provision like this for replacing~~ a bare URL ~~with a smarter link~~, see "Exceptions" below to remove the link from the report.		**If there's no way around a bare URL, see "Exceptions" below to remove the link from the report.
	*'''IW''': An external link (bare URL) ~~that~~ could be an [[Help:Editing#Interwiki_links\|interwiki link]]. Interwiki links are shorter and more resistant to rot. The suggested interwiki link markup will be given in the report. For foreign-language Wikipedia pages, you can add a language code, e.g. <nowiki>[[wp:de:Test]]</nowiki> for the German version of the page.		*'''IW''': This marks an external link (bare URL) to another wiki which could be an [[Help:Editing#Interwiki_links\|interwiki link]]. Interwiki links are shorter and more resistant to rot. The suggested interwiki link markup will be given in the report. For foreign-language Wikipedia pages, you can add a language code, e.g. <nowiki>[[wp:de:Test]]</nowiki> for the German version of the page.
	*'''(xxx)''': The HTTP response code (see reference [[/HTTP codes\|HERE]]).		*'''(xxx)''': The HTTP response code (see reference [[/HTTP codes\|HERE]]).
	*'''(000-xx)''': The Unix tool 'curl' did not get an HTTP response code~~, but instead returned this exit code (see~~ [[/Curl codes\|HERE]]). The most common by far is "000-28", a timeout.		*'''(000-xx)''': The response code from the Unix tool 'curl', for cases where it did not get an HTTP response code. See [[/Curl codes\|HERE]] for a list of codes. The most common curl error by far is "000-28", a timeout from an unresponsive site.

	===Exceptions===		===Exceptions===
	Some links simply ~~have to~~ be presented in an unconventional way which Val thinks is a problem. Some links return error codes but actually work fine. ~~These~~ links can be added to the [[/Exceptions\|exceptions list]] in order to hide them in future reports.		Some links simply must be presented in an unconventional way which Val thinks is a problem. Some links return error codes but actually work fine. Such links can be added to the [[/Exceptions\|exceptions list]] in order to hide them in future reports.

	==Coming features==		==Coming features==
	*It seems that many external links no longer display the content that they were intended to display. In many cases, web sites are silently redirecting the user to their main page without using the appropriate code that indicates the content was not found. Only visual inspections of the pages can catch these issues. Once we have dealt with the more obvious link issues, the screenshot feature in the script ~~will~~ be activated, which will allow us to ~~easily~~ confirm if the "OK" links are loading the intended page.		*It seems that many external links no longer display the content that they were intended to display. In many cases, web sites are silently redirecting the user to their main page without using the appropriate code that indicates the content was not found. Only visual inspections of the pages can catch these issues. Once we have dealt with the more obvious link issues, the screenshot feature in the script can be activated, which will allow us to confirm if the "OK" links are loading the intended page.

	*Over time, the exceptions list should be audited so it doesn't acquire [[wikt:cruft\|cruft]]. Val needs to ~~mention~~ URLs in the list that aren't in ~~extlinks.csv~~ anymore, ~~and~~ URLs which ~~returned different codes from what was expected~~.		*Over time, the exceptions list should be audited so it doesn't acquire [[wikt:cruft\|cruft]]. Val needs to report URLs in the list that aren't even in the external links table anymore, as well as URLs in the exceptions list which no longer return an error code.

	==Source code==		==Source code==
	The project is found [http://websvn.chrilly.net/listing.php?repname=Oni2&path=%2FValidate+External+Links%2F ~~here~~]. Along with the Bash script itself, you'll find documentation on how ~~exactly~~ to run ValExtLinks on your own computer as well as resources for contributing to the code.		The project is found [http://websvn.chrilly.net/listing.php?repname=Oni2&path=%2FValidate+External+Links%2F HERE]. Along with the Bash script itself, you'll find documentation on how to run ValExtLinks on your own computer as well as resources for contributing to the code.

	[[Category:Wiki Support]]		[[Category:Wiki Support]]