Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Current State of Financial Data Available to Perl

by justin423 (Sexton)
on Sep 25, 2022 at 18:35 UTC ( #11147125=perlquestion: print w/replies, xml ) Need Help??

justin423 has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to build a program that downloads the data on options on a handful of securities once a day after the market closes. There are a number of sites that provide an API for Python, but I am not seeing the same for Perl. the modules that would do this look very out of date (like Finance::QuoteOptions) It looks like yahoo and google removed their finance data download options. I know this is a longshot, but just thought I would ask here.
  • Comment on Current State of Financial Data Available to Perl

Replies are listed 'Best First'.
Re: Current State of Financial Data Available to Perl
by Your Mother (Archbishop) on Sep 25, 2022 at 20:05 UTC

    Iíve written a fair amount of code like this for Ally, Alpaca, AlphaVantage, Binance, Coinbase, IEX, Intrinio, Quodd, TDA, Tradier, and WorldTradingData (just scanning my git repos). I never released any because it was for personal use and either poorly generalized or half-done. Caveat: I have no clue what state the stuff is in. I havenít looked at any of it but the Binance stuff since 2019 and I donít know what I did or left undone; probably only 15% of what I cited was ever fully working because I was sampling whichever services were easiest to code for. Plenty is available for free but you generally need an account to talk to the APIs.

    That said, I am happy to share or discuss specific pieces or mechanics. Sorry I canít just open up my private repos. They would need a serious vetting of what I left in them and I have no time for that.

    More than once I have looked at the sample python code for services and adapted it to Perl. Often the documentation for services is incomplete, dated, or just wrong and being able to read working code shows how the developers managed to figure out what was left unsaid.

      ok, thanks for confirming my suspicions.
Re: Current State of Financial Data Available to Perl
by bliako (Monsignor) on Sep 26, 2022 at 12:43 UTC

    I resent the pessimism and I resent the mention of that dreaded python for any use really. As if it is the standard technology (standard it may has become, but technology it isn't). Albeit, my resentment is in a friendly way :)

    So, in short, do not despair.

    What stops you from getting a yummy, juicy json chunk from this endpoint of Yahoo's finance services: https://query1.finance.yahoo.com/v10/finance/quoteSummary/API?modules=financialData ???? Why can't you scrape that with Perl (assuming you want to)?

    In absence of a better link (API is also a stockmarket symbol, LOLz) this will get you started (you can safely ignore their recommended tools and use wget or curl from the CLI to start, and then LWP::UserAgent and JSON for cooking a super-duper, Perl stock scrapper and all that without parsing ANY html whatsoever): https://syncwith.com/yahoo-finance/yahoo-finance-api

    I do not want to take the fun from your hands so I will not supply any working code, but if you need more help, let us know,

    bw bliako

    Edit: and here is an endpoint providing endless pessimism: https://query1.finance.yahoo.com/v7/finance/quote?symbols=GBPUSD=X&fields=regularMarketPrice

Re: Current State of Financial Data Available to Perl
by harangzsolt33 (Hermit) on Sep 26, 2022 at 02:18 UTC
    If I had to do something like that, I would simply reverse engineer a site like BigCharts options chain and use Perl to extract the raw data from the HTML page source. See: https://bigcharts.marketwatch.com/quickchart/options.asp?symb=AAPL

    A bit more involved task would be to log into my Robinhood account and reverse engineer the options quote source and write an alternative code in JavaScript that downloads all the quotes I need OR write a program that automatically copies the content I need. In Windows, you can automate such things like mouse clicks and keyboard presses, so you can write a program either in VBScript or JavaScript that opens Internet Explorer, loads a site, enters username and password, waits, then clicks here and there... Copies either the page content or the page source and saves it to a file. then loads the next page and copies that one too. and so forth.. I have done stuff like that to download a list of all stock symbols and their names and CUSIP numbers from various sites. It ran for about 3 days non-stop.

    For a fun automation example, here is a JavaScript program that when you double-click it in Windows, it overwrites itself, then deletes itself sort of like a virus... by sending various key presses same as if somebody stole your keyboard. lol

    SELF = WScript.ScriptFullName; WshShell = WScript.CreateObject("WScript.Shell"); RUN("C:\\WINDOWS\\NOTEPAD.EXE " + SELF); WINFOCUS("notepad"); WAIT(500); PRESS("^{a}"); // CTRL+a (SELECT ALL) TYPE("WScript.Echo"); // START TYPING PRESS("{(}"); TYPE("'Muhahaha'"); PRESS("{)}"); TYPE(";\n"); // UNCOMMENT THESE FOR THIS SCRIPT TO ACTUALLY WORK: //PRESS("%F"); // ALT+F //PRESS("x"); // CHOOSE EXIT //PRESS("{ENTER}"); // PRESS ENTER TO SAVE CHANGES. SAY("HAHA"); ALERT("This is the self-destroying script. It's gone! :)"); function WINFOCUS(W) { WshShell.AppActivate(W); } function WAIT(MS) { WScript.Sleep(MS); } function RUN(CMD) { WshShell.Run(CMD, 9); } function SAY(TEXT) { var VOICE = WScript.CreateObject("SAPI.SpVoice"); + VOICE.Volume = 100; VOICE.Speak(TEXT); } function EXIT() { WScript.Quit(0); } function PRESS(KEYCODE) { WAIT(100); WshShell.SendKeys(KEYCODE); } function TYPE(TEXT) { var i, t; for (i = 0; i < TEXT.length; i++) { t += TEXT.charAt(i); WAIT(20); if (i > 8) { t = TEXT.slice(i); i = TEXT. +length; } PRESS(t); }} function ALERT(MSG) { WScript.Echo(MSG); }

    OKAY. Here is a program that actually downloads something useful. (MyPal is a port of PaleMoon web browser to Windows XP/7/10.) It reads stock symbol list from a file and then downloads an info page for each symbol, then copies the details into a text file and goes on to the next one. (Sorry, this is Windows JavaScript, not Perl, so it's slightly off-topic. But it's just to show you how to automate a very long robotic job on Windows and make it completely unattended self-processing.)

    SOURCE = "http://thestockmarketwatch.com/stock/?stock=$$$"; WEB_BROWSER = "C:\\BIN\\MYPAL\\mypal.exe"; WEB_BROWSER_NAME = "mypal"; SELF = WScript.ScriptFullName; CURRENT_DIRECTORY = cut(SELF, "\\", 0x10010); WshShell = WScript.CreateObject("WScript.Shell"); WordObj = WScript.CreateObject("Word.Application"); // WE USE MS-WORD +ONLY TO READ NUMLOCK STATE. ABC = "\n.ABCDEFGHIJKLMNOPQRSTUVWXYZ"; ALERT("NUMLOCK ON = RUN SCRIPT\n\nNUMLOCK OFF = STOP SCRIPT"); PAGE_SAVED = 0; INPUT_FILE = CURRENT_DIRECTORY + "\\SYMBOL_LIST.TXT"; OUTPUT_FILE = CURRENT_DIRECTORY + "\\ALLCOMPETITORS.TXT"; SYMBOLS = ReadFile(INPUT_FILE).toUpperCase(); SYMBOLS = SYMBOLS.split("\r\n"); ALERT("ABOUT TO DOWNLOAD " + SYMBOLS.length + " STOCK DATA."); for (i = 0; i < SYMBOLS.length; i++) { CURRENT_SYMBOL = Trim(SYMBOLS[i]); URL = SOURCE; URL = URL.split("$$$").join(CURRENT_SYMBOL); if (i == 0) { RUN(QUOTE(WEB_BROWSER) + " " + QUOTE(URL)); WAIT(22); WINFOCUS(WEB_BROWSER_NAME); } else { PRESS(URL); WAIT(1); PRESS("{ENTER}") WAIT(18); } PRESS("^{a}"); // SELECT ALL WAIT(1.5); PRESS("^{INSERT}"); // COPY WAIT(3); DATA = ReadClipboardText(); Process_and_Save(DATA); WAIT(1); WINFOCUS(WEB_BROWSER_NAME); PRESS("+{TAB}"); // SHIFT+TAB WAIT(0.5); PRESS("{TAB}"); // TAB WAIT(0.5); } ALERT("DONE."); EXIT(0); ////////////////////////////////////////////////// function Process_and_Save(DATA) { var ALLOWED = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0 +123456789 \t\n_.,<>(){}[]-+=\\|'\":;?/!`~@#$%^&*"; DATA = tr(DATA, ALLOWED); if (DATA.indexOf("Symbol Lookup") < 0) return; if (Trim(Between(DATA, "Company Profile", "Home")) == "") { SAVE_LIST = CURRENT_SYMBOL + " -\r\n\r\n"; AppendFile(OUTPUT_FILE, SAVE_LIST); return; } DATA = cut(DATA, "Learn", 0x101); DATA = cut(DATA, "Home", 0x10110); var PROFILE = cut(DATA, "Company Profile", 0x10001).split("\n").join +(";"); var COMPETITORS = cut($a, "100% secure: your email is never shared, +and you may opt out at any time.", 1).split("\n").join(";"); var NUMBERS = Between(DATA, "Learn", "1 day").split("\n").join(";"); var SAVE_LIST = [CURRENT_SYMBOL, PROFILE, COMPETITORS, NUMBERS].join +("|") + "\r\n\r\n"; AppendFile(OUTPUT_FILE, SAVE_LIST); } ////////////////////////////////////////////////// // This function extracts a section from string S that lies between st +ring L and R. // Returns an empty string if either L or R is not found. function Between(S, L, R, SKIP) { if (typeof(SKIP) === "undefined") SKIP = 1; var p1 = 0; for (var i = 0; i < SKIP; i++) { p1 = S.indexOf(L, p1); if (p1 < 0) return ""; p1 += L.length; } var p2 = S.indexOf(R, p1); if (p2 <= p1) return ""; return Trim(S.substring(p1, p2)); } ////////////////////////////////////////////////// // // F U N C T I O N L I B R A R Y // function Abort_Script_If_Numlock_Is_OFF() { if (GetNumlockState() == 0 +) EXIT(0); } function WINFOCUS(W) { Abort_Script_If_Numlock_Is_OFF(); WshShell.AppA +ctivate(W); } function WAIT(MS) { Abort_Script_If_Numlock_Is_OFF(); MS *= 1000; WScr +ipt.Sleep(MS); Abort_Script_If_Numlock_Is_OFF(); } function RUN(CMD) { Abort_Script_If_Numlock_Is_OFF(); WshShell.Run(CMD +, 9); } function SAY(TEXT) { Abort_Script_If_Numlock_Is_OFF(); var VOICE = WSc +ript.CreateObject("SAPI.SpVoice"); VOICE.Volume = 100; VOICE.Speak(TE +XT); } function EXIT(E) { WordObj.Quit(); if (typeof(E) === "undefined") E = +0; WScript.Quit(E); } function QUIT(E) { WordObj.Quit(); if (typeof(E) === "undefined") E = +0; WScript.Quit(E); } function PRESS(KEYCODE) { WAIT(0.1); WshShell.SendKeys(KEYCODE); Abort +_Script_If_Numlock_Is_OFF(); } function ALERT(MSG) { WScript.Echo(MSG); } function QUOTE(S) { return '"' + S + '"'; } function DEBUG(S) { ALERT("DEBUG:|" + S + "|"); } ////////////////////////////////////////////////// // This function positions our window in the center of the screen. function CENTER(W, H) { self.resizeTo(W, H); self.moveTo(Math.round((s +creen.width - W) / 2), Math.round((screen.height - H) / 2)); } ////////////////////////////////////////////////// // Creates and overwrites a text file and with a string. // Return 0 on success or 1 if an error occurred. // Global variable $a will hold an error message or the // word "SUCCESS" if there were no errors. // Usage: INTEGER = CreateFile(FILENAME, STRING) // function CreateFile(FILENAME, STRING) { $a = ""; var inASCII = 0; var inUNICODE = 1; var Overwrite = 1; var Do_Not_Overwrite = 0; try { var FSO = new ActiveXObject("Scripting.FileSystemObject"); var F = FSO.CreateTextFile(FILENAME, Overwrite, inASCII); F.Write(STRING); F.Close(); $a = "SUCCESS"; return 0; } catch (e) { $a = e.message; } return 1; } // Deletes a file. Return 0 on success or an error message. function DeleteFile(FILENAME, DELETE_READONLY) { try { FSO.DeleteFile( +FILENAME, DELETE_READONLY); return 0; } catch(e) { return e.message; +} } ////////////////////////////////////////////////// function TYPE(TEXT) { Abort_Script_If_Numlock_Is_OFF(); var i, t; for (i = 0; i < TEXT.length; i++) { t = TEXT.charAt(i); WAIT(0.1); if (i > 5) { t = TEXT.slice(i); i = TEXT.length; } PRESS(t); } } ////////////////////////////////////////////////// /* Most ASCII characters can be represented by the character itself. E.g, the key sequence FRED can be represented by "FRED". Some special keys, such as the control keys, function keys etc are enc +oded in a string enclosed by {braces} See the table below Key/Character SendKey Description ~ {~} Send a tilde (~) ! {!} Send an exclamation point (!) ^ {^} Send a caret (^) + {+} Send a plus sign (+) Backspace {BACKSPACE} or {BKSP} or {BS} Send a Backspace keyst +roke Break {BREAK} Send a Break keystroke Caps Lock {CAPSLOCK} Press the Caps Lock Key (toggle on or off +) Clear {CLEAR} Clear the field Delete {DELETE} or {DEL} Send a Delete keystroke Insert {INSERT} or {INS} Send an Insert keystroke Cursor control arrows {LEFT} / {RIGHT} / {UP} / {DOWN} Send a +Left/Right/Up/Down Arrow End {END} Send an End keystroke Enter {ENTER} or ~ Send an Enter keystroke Escape {ESCAPE} Send an Esc keystroke F1 through F16 {F1} through {F16} Send a Function keystroke Help {HELP} Send a Help keystroke Home {HOME} Send a Home keystroke Numlock {NUMLOCK} Send a Num Lock keystroke Page Down Page Up {PGDN} {PGUP} Send a Page Down or Page Up keystroke Print Screen {PRTSC} Send a Print Screen keystroke Scroll lock {SCROLLLOCK} Press the Scroll lock Key (toggle on +or off) TAB {TAB} Send a TAB keystroke To specify keys combined with any combination of SHIFT, CTRL, and ALT +keys, precede the key code with one or more of the following: For SHIFT prefix with + For CTRL prefix with ^ For ALT prefix with % Example ' Open notepad Set WshShell = WScript.CreateObject("WScript.Shell") WshShell.Run "notepad.exe", 9 ' Give Notepad time to load WScript.Sleep 500 ' Type in Hello World WshShell.SendKeys "Hello World!" WshShell.SendKeys "{ENTER}" ' Add the date WshShell.SendKeys "{F5}" */ ////////////////////////////////////////////////// // // Returns the entire contents of a text or binary file. // Usage: STRING = ReadFile(FILENAME) // function ReadFile(FILENAME) { var F, FSO, DATA = ""; try { FSO = new ActiveXObject("Scripting.FileSystemObject"); F = FSO.OpenTextFile(FILENAME, 1); DATA = F.ReadAll(); F.Close(); return DATA; } catch (e) {} return ""; } ////////////////////////////////////////////////// // This function splits string into two parts // along the first occurrence of substring. // Returns the first part if CMD is 0x10. // Returns the second part if CMD is 1. // If substring is not found, returns the original // string if CMD is 0x100. Ignores case when // CMD is 0x1000. Starts searching from end of // string when CMD is 0x10000. // function cut(STR, SUB, CMD) { if (typeof(CMD) === "undefined") CMD = 0x111; STR += ""; SUB += ""; var P; if (CMD & 0x1000) P = (CMD & 0x10000) ? STR.toUpperCase().lastIndexOf(SUB) : STR.toU +pperCase().indexOf(SUB); else P = (CMD & 0x10000) ? STR.lastIndexOf(SUB) : STR.indexOf(SUB); $a = $b = ""; if (P < 0) return (CMD & 256) ? STR : ""; $a = STR.substr(0, P); $b = STR.slice(P + SUB.length); return (CMD & 16 ? $a : '') + (CMD & 1 ? $b : ''); } ////////////////////////////////////////////////// // v2021.3.15 // This function removes all characters from STR // that do not appear anywhere in CHARSET, forcing // string to be only made up of the given characters. // Example: tr("cabbage cake", " abc") --> "cabba ca" // [This function has not been optimized yet.] // Usage: STRING = tr(STRING, STRING) // function tr(STR, CHARSET) { STR += ""; STR = STR.split(""); for (var i = 0; i < STR.length; i++) if (CHARSET.indexOf(STR[i]) < 0) STR[i] = ""; return STR.join(""); } ////////////////////////////////////////////////// // v2021.3.15 // This function returns 1 when Numlock is ON. // Returns 0 when Numlock is OFF. // Usage: INTEGER = GetNumlockState() // function GetNumlockState() { return (WordObj.NumLock) & 1; } ////////////////////////////////////////////////// // v2021.3.15 // This function returns 1 if the file exists. // Returns 0 if the file doesn't exist. // Usage: INTEGER = FileExists(FILENAME) // function FileExists(FILENAME) { var FSO, E = 0; try { FSO = new ActiveXObject("Scripting.FileSystemObject"); E = FSO.FileExists(FILENAME); } catch (e) {} return E; } ////////////////////////////////////////////////// // v2021.3.16 // This function copies all text from the clipboard. // This is done by creating an Internet Explorer // document in the background that contains a form // and a textbox for input. After pasting whatever // is on the clipboard into the textarea, we can // reads its contents and return the string. // This takes about 200ms. // // Usage: STRING = ReadClipboardText() // function ReadClipboardText() { var MSIE = WScript.CreateObject("InternetExplorer.Application"); MSIE.Visible = 0; MSIE.ToolBar = 0; MSIE.StatusBar = 0; MSIE.FullScreen = 1; MSIE.Navigate("about:blank"); /* MSIE.RegisterAsDropTarget = 0; WScript.Sleep(100); MSIE.Width = 600; MSIE.Height = 400; MSIE.Left = (MSIE.document.parentWindow.screen.availWidth - MSIE.Wid +th) / 2; MSIE.Top = (MSIE.document.parentWindow.screen.availHeight - MSIE.Hei +ght) / 2; */ var HTML = "<HTML><HEAD><TITLE></TITLE></HEAD><BODY onLoad='Init();' + SCROLL=NO><FORM NAME=MAIN><TEXTAREA NAME=INPUT COLS=20 ROWS=5></TEXT +AREA><SCRIPT> function Init() { document.MAIN.INPUT.focus(); } </SCRI +PT>"; WScript.Sleep(100); MSIE.Document.open(); MSIE.Document.write(HTML); MSIE.Document.close(); WScript.Sleep(100); PRESS("+{INSERT}"); WScript.Sleep(150); var TEXT = MSIE.Document.MAIN.INPUT.value + ""; WScript.Sleep(50); MSIE.Quit(); return TEXT; } ////////////////////////////////////////////////// function uc(S) { return S.toUpperCase(); } function lc(S) { return S.toLowerCase(); } ////////////////////////////////////////////////// // v2021.3.16 // This function adds a string to the end of a // file in ASCII mode. Return 0 on success or // returns 1 if something went wrong. The global // variable $a will hold a text string "SUCCESS" // if everything went well or an error message. // Usage: INTEGER = AppendFile(FILENAME, CONTENT) // function AppendFile(FILENAME, STRING) { $a = ""; var ForReading = 2; var ForWriting = 4; var ForAppending = 8; var inASCII_Format = 0; var inUNICODE_Format = -1; var Create_If_File_Doesnt_Exist = 1; var Do_Not_Create_If_File_Doesnt_Exist = 0; try { var FSO = new ActiveXObject("Scripting.FileSystemObject"); var F = FSO.OpenTextFile(FILENAME, ForAppending, Create_If_File_Do +esnt_Exist, inASCII_Format); F.Write(STRING); F.Close(); return 0; } catch (e) { $a = e.message; } return 1; } ////////////////////////////////////////////////// // // This function removes whitespace before and after // text T and returns a new string. // Usage: STRING = Trim(STRING) // function Trim(T) { T += ""; var i, j, x; for (x = i = j = 0, i--; x < T.length; x++) if (T.charCodeAt(x) > 32) { if (i < 0) i = x; j = x + 1; } return T.slice(i, j); } ////////////////////////////////////////////////// // // This function is just like Trim() except it // removes characters specified in SUBSTR. // Usage: STRING = TrimChar(STRING, SUBSTR) // function TrimChar(STR, CHARLIST) { STR += ""; var L = STR.length, START = 0, LAST = 0; while (L--) { if (CHARLIST.indexOf(STR.substr(L, 1)) < 0) { START = L; if (LAST == 0) LAST = L + 1; } } return STR.substring(START, LAST); } ////////////////////////////////////////////////// // // This function strips html tags from a string // and returns the text portion only. // Usage: STRING = StripHTML(STRING) function StripHTML(S) { var i, c, SKIP = 0, OUTPUT = []; for (i = 0; i < S.length; i++) { c = S.charCodeAt(i); if (c == 60) SKIP = 1; if (SKIP == 0) OUTPUT.push(S.substr(i, 1)); if (c == 62) SKIP = 0; } OUTPUT = OUTPUT.join(""); OUTPUT = OUTPUT.replace(/\&nbsp;/g, " "); OUTPUT = OUTPUT.replace(/\&amp;/g, "&"); OUTPUT = OUTPUT.replace(/\t/g, " "); OUTPUT = OUTPUT.replace(/ /g, " "); return OUTPUT; } ////////////////////////////////////////////////// // This function sorts array A and removes duplicate elements. function RemoveDuplicates(A) { A.sort(); var LINE, i, j = 0, prev = ""; for (i = 0; i < A.length; i++) { LINE = Trim(A[i]); if (LINE == prev) LINE = ""; else prev = LINE; if (LINE.length) { if (j < i) A[j] = LINE; j++; } } A.length = j; } ////////////////////////////////////////////////// function uc(S) { return S.toUpperCase(); } function lc(S) { return S.toLowerCase(); } ////////////////////////////////////////////////// // v2019.12.9 // Returns one or more words from a string. // The string is treated as a list of words separated // by whitespace. In this case, a "whitespace" is any // character whose ASCII value is less than 33. This // includes new line characters, tab, space, null, etc. // PTR tells which word to grab starting with 1. // If PTR is 3, the third word is returned. // If PTR is not specified, the default value is 1. // COUNT tells how many words to return. Default is 1. // When COUNT has a negative value, returns every word // from PTR all the way to the end of the string. // The words in the return value will always be separated // by a space character regardless of how many spaces or // tabs were between them in the input string. // // Usage: STRING = GetWord(STRING, [PTR, [COUNT]]) // function GetWord(S, PTR, COUNT) { S += ""; if (typeof(PTR) === "undefined") PTR = 1; if (typeof(COUNT) === "undefined") COUNT = 1; if (S.length == 0 || COUNT == 0 || PTR >= S.length) return ""; if (PTR <= 0) PTR = 1; var i, START = -1, OUTPUT = []; for (i = 0; i <= S.length; i++) { if (S.charCodeAt(i) > 32) { if (START < 0) START = i; continue; } if (START >= 0) { if (PTR-- < 2) { OUTPUT.push(S.substring(START, i)); if (COUNT-- == 1) break; } START = -1; } } return OUTPUT.join(" "); }

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11147125]
Approved by davies
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (3)
As of 2022-12-01 19:07 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Notices?