2021-11-26 pyautogui cooperates with lightning simulator to realize the automation of mobile app check-in and answer questions

Portal of basic command

Extended portal

With basic commands, you can perform many operations of finding a map, clicking and selecting points , So what does this blog record ？

First ,Python and PC The end fit is very good , Omnipotent language is not blowing . I use PYAUTOGUI The main purpose is to cooperate with the mobile phone simulator to realize APP Sign in and answer questions . The simulator selected this time is the lightning simulator , A lot of map finding and click commands are used here , After all, manual check-in is also to find pictures and click , In fact, the logic is the same .

1. First , Pay attention to the resolution setting of the simulation , Once you are sure, don't change , Take a screenshot at this resolution , Then it's easy to lose it after changing the picture .
   
2. Prepare several libraries pyautogui, There are also shear board read and write modules pyperclip, as well as opencv Carry out certain map similarity control pip install opencv-python

import pyperclip
pyperclip.copy('123') #  Write clipboard 
dd=pyperclip.paste() # obtain 

3. Wrap a function to find the graph , A large number of full screen map finding must be reused

import pyautogui,time
def find_bmp_all(path): # Pass in the picture location and name 
    a=list(pyautogui.locateAllOnScreen(path,confidence=0.9))# Find all the graphs and return a list 
    print(a)
    if len(a)==1:# Find a situation 
        b=pyautogui.center(a[0])  # Determine the center point 
        print(b)
        pyautogui.moveTo(*b,duration=0.25)# Move the mouse to the center 
        return b  # Return the coordinates of the central position 
    else:  
        return []# Null values will be returned if no or multiple items are found 

Turn the question answering process into a list to realize sequential operation

def lianxi():

    for i in ['xuexi.png', 'xuexi2.png', 'dati.png', 'lianxi.png', 'dianbengdui.png',
              'zhengque.png', 'querendaan.png', 'xiayiti.png', 'zhengque.png',
              'querendaan.png', 'xiayiti.png', 'jiesulianxi.png',
              'cha.png', 'cha.png']:
        xy = find_bmp_all(i)# Pass in pictures in order to find , Because our options are fixed A Options so , The process is very rough. You can choose directly without judging .
        if xy:# If you find it , Just click on the coordinates found 
            pyautogui.click(*xy)
        time.sleep(2)

What if you need to find the answer , Ideas
1、 Copy what you want to get to the clipboard , Then the shear board reads , But this time, I found that the things in the simulator and the shear board library were not working properly , Since it's abnormal, I won't study it deeply , My goal APP Click answer analysis to directly display the answer, for example right key ：1、2、3、4 Such a structure , But you should check the answer he gave again . that ~
2、 The idea is changed to click the answer analysis first , Full screen search （ right key ：） This a few word , Then determine the general coordinates of the answer according to the found coordinates , Then the small area map finding is carried out four times of table finding 1,2,3,4 Add what you find to a list , Then use one FOR Loop out the found list and find the corresponding ABCD Item and check .

def qiandao():
    for i in ['xuexi.png', 'xuexi2.png','wode.png' ,'chakanjiexi.png']: # Go to the fixed check-in page first and click the answer analysis 
        xy = find_bmp_all(i)
        if xy:
            pyautogui.click(*xy)
        time.sleep(2)
    time.sleep(4)# Have a good sleep 
    xy = list(pyautogui.locateAllOnScreen('qiandaodaan.png', confidence=0.9))# look for   right key ：  This position 
    print('123213',xy[0])
    num_all=[]
    for i in ['1.png','2.png','3.png','4.png']: # Circular small area search 1,2,3,4
        yes_no= list(pyautogui.locateAllOnScreen(i,confidence=0.9, region=(xy[0][0], xy[0][1]-50, 300,100)))
        print(yes_no)
        if yes_no :# Add to the list if you have 
            num_all.append(int(i[0]))
    print(num_all)
    pyautogui.moveRel(0,-300) # Move the mouse up 300
    time.sleep(0.5)
    pyautogui.scroll(300) # Roll up and down 300, Put all the answers and options on the screen and don't block them 
    time.sleep(1)
    for i in num_all:# Select the pictures and times to search according to the list 
        xy_0=find_bmp_all('xuanze%d.png'%i)
        print(xy_0)
        if xy_0:
            pyautogui.click(*xy_0)
            time.sleep(1)
        else:
            print('meizhaodao')

    xy_0 = find_bmp_all('qiandaoquerendaan.png') # Choose a point to confirm the answer 
    print(xy_0)
    time.sleep(1)
    if xy_0:
        pyautogui.click(*xy_0)
        time.sleep(1)
    xy_0 = find_bmp_all('qiandaoxiayiti.png')# Click on the next question 
    print(xy_0)
    time.sleep(1)
    if xy_0:# Found click 
        pyautogui.click(*xy_0)
        time.sleep(1)
    else :# If you can't find it , That should be no problem , Find the finish button 
        xy_0 = find_bmp_all('qiandaowancheng.png')
        if xy_0:
            pyautogui.click(*xy_0)

The following is the screenshot icon used .

summary , This is a usage record , In fact, the technical content is not high, but logic and business are closely combined , In line with the principle of being able to use, I made this program simply and roughly . Making programs takes less time than blogging , The first reason for posting records is that I need the function of automatically answering questions and punching cards on my mobile phone . Two is 、 Creatively use screenshots to identify the options for finding answers , At least not used OCR Recognition also completes the recognition function . The third is 、 This kind of development program is very messy , It's better to record .